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ABSTRACT 


In  a  joint  effort,  the  U.S.  Navy  and  the  Air  Force  have  tested  under  operational  conditions  a  series 
of  inp-oved  techniques  developed  under  the  Air  Force  Integrated  Maintenance  Information  System  (IMS) 
program  for  delivering  maintenance  Technical  Information  to  squadron  Technicians.  These  inprovements 
included  use  of  a  Portable  Maintenance  Aid  (PMA)  for  Technical  Information  display.  In  this  test,  canried 
out  in  an  F/A-18  fighter  squadron  at  Marine  Corps  Air  Station,  Beaufort,  South  Carolina,  each  of  16 
Technicians  performed  six  Fault-Isolation  tasks,  three  supported  by  an  Interactive  Electronic  Technical 
Mmual  (lETM)  displayed  on  a  PMA,  and  three  sipported  by  a  conventional  paper-based  Work  Package 
(WP)  Technical  Manual.  This  report  presents  a  comparison  of  Technician  performance  sipported  by  the 
lETM/PMA  combination  with  performance  using  conventional  paper  Technical  Manuals.  The  performance 
data  collected  during  the  test  showed  considerable  reduction  in  performance  times  for  conplex,  multiple 
Faults  Isolations,  and  reduction  of  maintenance  errors,  when  Technicians  used  the  lETM/PMA 
combinatioa  Also,  Technician-preference  data  based  on  questionnaires  showed  strong  sipport  for 
virtually  all  lETMPMA  features.  Technicians  also  made  suggestions  for  inproving  the  lETM  and  the 
PMA  The  test  results  have  been  provided  to  both  Services  as  one  basis  for  evaluating  and  improving 
these  IMS  technologies;  specifically,  (1)  the  specially  developed  PMA;  (2)  preparation  of  the  lETM 
Technical  Information  by  automated  methods  based  on  initial  construction  of  an  lETM  data  base 
(lETMDB);  (3)  The  IMS  Presentation  System  (PS);  (4)  on-ground  Fault  Isolation  using  data  from  aircraft 
Built-In  Test  Equipment  (BITE)  coipled  directly  to  the  PMA  throu^  a  1553  bus;  (5)  the  IMS  Diagnostic 
Module  (DN^,  and  (6)  the  IMS  Human  Conputer  Interface  (HQ)  module.  The  lETMDB  and  the  lETM 
Technical  Information  were  constructed  generally  in  accordance  with  the  new  DOD  Specifications 
MIl^D-87269  and  MILrM-87268,  respectively. 


ADMNISTRATIVE  INFORMATION 


The  work  presented  in  this  report  was  accomplished  under  the  direction 
of  the  Carderock  Division,  Naval  Surface  Warfare  Center  (formerly  the 
David  Taylor  Research  Center)  under  Job  Order  JO  1-1223-103  funded 
by  NAVAIR  41 1  under  AIRTASK  A41141 14-0104-141 1400001. 


2 


I 

I 

I 


i/ 


1.0  TEST  SUMMARY 

1.1  TEST  OVERVIEW 

In  a  joint  effort,  tlifc  Navy  and  the  Air  Force  have  carried  out  an  operational  test  of  a  series  of 
technological  inpovements  developed  by  the  US  Air  Force  under  its  Integrated  ^  '  intenance  Information 
System  (IMIS)  Program  for  delivering  maintenance  Technical  Information  '  technicians.  These 
innovations  consisted  of; 

a.  Creation  of  maintenance  Technical  Information  throu^  use  of  an  Interactive  Electronic  Technical 
Manual  Data  Base  (lETMDB)  meeting  the  general  requirements  of  the  newly  developed 
MIL-D-87269;  Data  Base,  Revisable  Interact  Electronic  Technical  Manuals,  for  the  Support 
of,  20  Nov  1992  (ref  1). 

b.  Conversion  of  the  Technical  Information  in  the  lETMDB  to  a  form  suitable  for  viewing  by  the 
end  user  on  a  Portable  Maintenance  Aid  (PMA)  through  use  of  the  IMIS  PS  (Presentation  System) 
software  housed  in  the  PMA  itself  (Note  that  the  IMIS  term  PMA,  which  will  be  used 
throu^out  this  report,  is  synonymous  with  the  term  Portable  Electronic  Delivery  Device  (PEDD) 
used  in  the  US.  Naval  Air  Systems  Command.) 

c.  The  field  use  of  PMAs  for  Fault  Isolation. 

d  The  use  of  direct  interaction  between  airraaft  Built-In  Test  Equipment  (BITE)  and  the  PMA, 
through  a  1553  bus  to  facilitate  on-ground  Fault  Isolatioa  The  on-board  data  were  integrated  in 
the  PMA  by  the  IMIS  Diagnostic  Module  (DN^. 

e.  The  use  of  the  HQ  (Human  Conputer  Interface)  model  of  IMIS  for  presentation  of  the  Technical 
Information,  which  is  generally  in  accordance  with  the  newly  issued  MIL-M-87268;  Manuals, 
Interactive  Electronic  Technical:  General  Content,  Style,  Format,  and  User-Interaction 
Requirements,  20  Nov  1992  (ref  2). 

Each  Technician  performed  six  Fault-Isolation  tasks,  three  supported  by  an  Interactive  Electronic 
Technical  Manual  displayed  on  a  develcpmental  Portable  Maintenance  Aid  (lETMPMA)  and  three 
sipported  by  the  squadron's  conventional  paper-based  Work  Package  (WP)  Technical  Manuals,  The 
lETMPMA  combination  represented  the  most  ip-to-date  technology  of  the  Air  Force's  Integrated 
Maintenance  Information  System  (IMIS)  Program  Tests  were  made  using  the  Flight  Control  System  of 
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the  Navy  F/A-18  aircraft.  The  test  facility  and  test  personnel  were  provided  by  the  USMC  Filter 
Squadron  VMFA-312,  Marine  ODrps  Air  Station,  Beaufort,  SC.  After  a  Pretest  (or  Pilot-test)  phase 
designed  to  identify  and  resolve  logistics,  technical,  or  other  problems,  the  Test  was  conducted  during  a 
three-week  period  from  31  May  1992  to  19  June  1992. 

1.2  PREPARATION  OF  TEST  MATERIALS 

The  lEIMs  used  were  based  onlEIMDB  material  (F/A-18  maintenance  data)  created  by  McDonnell 
Douglas  Aircraft  Conpiny  (MCAIR).  The  Technical  Information  in  tlte  lETMDB  was  extracted  and 
converted  to  EETM  format  by  IMS  Presentation  System  software  hosted  in  tlie  PMAs,  wWch  were 
specifically  constructed  for  this  Test  by  the  Air  Force  Armstrong  Laboratory,  Human  Resources 
Directorate  (AL/HRGO).  The  Carderock  Division,  Naval  Surface  Warfare  Center,  monitored  the  authoring 
process  and  the  resulting  lETM  product.  The  extraction,  cotr^iling,  and  formatting  processes  required 
were  accorr^lished  by  a  largely  automated  IMS  technique  called  the  Presentation  System  (PS). 


1.3  TEST  OBJECTIVE 

The  objective  of  the  test  was  to  evaluate  the  effectiveness  of  several  IMS  modules  developed  for  the 
preparation  and  display  of  lEIMs  as  conpared  with  paper-based  Technical  Manuals.  Specifically,  the 
IMS  modules  evaluated  included; 

a  A  data  base  prepared  for  the  F/A- 1 8  in  accordance  with  the  USAF  Content  Data  Model  (CDM), 
a  technique  which  forms  the  basis  of  ref  1. 

b.  A  new  li^itweight  Portable  Maintenance  Aid  (PMA)  built  by  AL/HRGO  under  the  IMS  program. 

c.  Inprovement  in  Fault-Isolation  effectiveness  resulting  from  direct  interaction  between  the  PMA 
and  the  F/A-18's  Built-In  Test  capabilities  via  the  1S53  multiplex  bus  interfaced  with  the  IMS 
Diagnostics  Module  (DM). 

d.  New  techniques  for  developing  and  displaying  troubleshooting  instructions,  i.e.,  instructions 
selected  dynamically  by  a  presentation  system  hosted  in  the  PMA,  based  on  previous  test 
outcomes  and  test-candidate  characteristics  siK;h  as  times-to-test  and  failure  rates  of  the  affected 
components. 


4 


e.  Conputer-assisted  conpletion  of  maintenance  records  [e.g.,  Visual  Information  Display 
System/Maintenance  Action  Form  (VIDS/MAF),  the  NAVAIR  maintenance  control  and  reporting 
form]. 


1.4  TEST  STRUCTURE 

Hie  test  conqiared  performances  of  Technicians  working  with  the  lETM/PMA  combination  to 
performances  of  Technicians  working  with  the  &dstingF/A-18  pc^scr-based  Work  Package  (WP)  Technical 
Manuals.  Sixteen  Mirine  Corps  Technicians  (with  Electrician  or  Communicadon/Radar/Navigarion  MOSs) 
performed  the  work.  Half  of  the  Technicians  were  experienced  and  half,  inexperienced.  After  an  initial 
indoctrination  session,  each  Technician  was  asked  to  perform  six  FaulMsolation  tasks,  three  guided  by 
lEIMPMA-presented  Tl  and  three  guided  by  paper-based  F/A-18  work-package  TMs  (i.e.,  a  total  of  96 
individual  tests),  Each  of  the  sbc  Fault-Isolation  tasks  was  divided  into  a  number  of  sequential  Intervals. 
The  faults  were  of  three  significantly  difterent  types. 

Data  collectors  recorded  performance  times  and  errors  for  each  Interval  of  the  participants' 
performances,  and  solicited  Technicians'  opinions  as  to  effectiveness  of  various  features  of  the  lETM/PMA 
combination. 


1.5  TEST  RESULTS 

lETM  technological  innovations  tested  in  the  field  worked  satisfactorily  in  all  cases.  Technicians 
were  able  to  perform  the  troubleshooting  tasks  assigned  in  a  shorter  time  with  the  lETMTMA 
combination  than  with  the  existing  paper-based  Technical  Manuals  (with  isolated  exceptions). 
Perfomiance  benefits  were  particularly  noticeable  with  regard  to  ineiqierienced  Technicians,  whose 
performance  was  brought  to  a  level  cqiproximating  that  of  the  ejqierienced  Technicians,  a  result  indicating 
that  the  advent  oflElMs  into  the  squadrons  could  significantly  reduce  the  requirements  for  training  and 
could  result  in  a  more  effective  use  of  inexperienced  maintenance  Technicians.  Fault-Isolation  errors  were 
few.  All  data  are  presented  in  detail  in  section  4  and  evaluated  in  section  5. 

In  response  to  questionnaire  based  interviews.  Technicians; 

•  Showed  preference  for  \’irtually  all  of  the  lETM/PMA  features,  and 
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Provided  a  number  of  suggestions  for  further  enhancement  of  the  lETMPMA  combination. 


1.6  TEST  CX)Na.USIONS 

All  phases  of  the  Joint  Navy/AF  F/A-18  tests  are  considered  to  have  been  successful  in  that  they; 

a.  Showed  that  the  extensive  automation  applied  to  procedures  for  creating  and  displaying  lETMs 
on  PMAs  produced  inpovements  in  Technician  performance  (over  pcrfonnance  with  paper  TMs) 
equivalent  to  that  produced  by  IFTMs  which  had  been  prepared  entirely  by  a  human  author. 

b.  Provided  detailed  guidance  for  inpovement  of  the  IMIS  developmental  procedures  and  modules 
tested,  both  as  a  result  of  the  quantitative  test  data  (e.g.,  performance  times)  and  as  a  result  of  an 
extensive  set  of  Technicians'  evaluations. 


1.7  STRUCTUREOFTESTREPORT 

Section  2  presents  background  information  leading  to  the  Test.  Section  3  describes  the  Test 
methodology.  Section  4  presents  the  results  attained  from  the  tests,  particularly  performance  time  and 
performance  errors.  Section  5  discusses  the  significance  of  these  results.  Section  6  presents  Conclusions 
and  Recommendations,  i^pendix  A  contains  a  detailed  summary  of  Technicians'  Comments  concerning 
the  lETM/PMA  combinations,  y^spendixes  B  through  E  show  the  forms  used  by  test  personnel  to  obtain 
Technician  Responses,  y^jpendix  F  presents  san^le  lElMPMA  frames. 
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2.0  TEST  BACKGROUND 


2.1  BACKGROUND 


2.1.1  The  Computer-aided  Acquisition  and  Logistic  Support  (CALS)  Initiative 

Under  the  CALS  Initiative,  the  Department  of  Defense  is  moving  to  apply  integrated  conpAer 
technology  to  siqjport  the  development  and  maintenance  of  its  weapon  systems.  In  siq>port  of  this  thrust, 
the  USAF  Armstrong  Laboratory,  Human  Resources  Directorate  (AL/HRGO)  and  the  Caiderock  Division 
of  the  Naval  Surface  Warfare  Center  (CDNSWC)  have  been  performing  coordinated  RDT&E  to  inpove 
the  presentation  of  Technical  Information  to  enlisted  Technicians  engaged  in  maintenance  of  weapon 
systems  throu^  the  use  of  interactive  electronic  display. 

2.1.2  lEIM  Development  Research  Hlstoiy 

Since  the  mid-1960s,  the  military  Services  have  sought  to  improve  the  overall  quality,  management, 
and  delivery  of  military  Technical  Information  in  all  of  its  aspects.  Initial  efforts  concentrated  on 
developing  inpoved  paper  Technical  Manuals.  Since  the  early  1980s,  the  emphasis  of  the  effort  has 
shifted  to  the  application  of  computer  technology  for  the  storage,  control,  and  presentation  of  maintenance, 
system-operation,  training,  and  other  forms  of  logistic-sipport  Technical  Informatioa  All  Services  have 
established  on-going  programs  to  develop  and  pply  this  technology.  Table  1  lists  the  major  operational 
tests  of  these  endeavors  to  date.  (See  References  for  complete  citations). 

The  advantages  and  deficiencies  of  the  H  preparation  and  delivery  qproaches  noted  during  these 
tests,  together  with  recent  technological  advances,  have  provided  the  basis  for  developing  the  next 
generation  of  electronic  delivery  systems.  Most  recently,  AL/HRGO  and  CDNSWC  have  collaborated 
on  the  field  test  of  an  lETM/PMA  combination  using  as  a  test  bed  the  Fli^t  Control  System  (PCS)  of 
the  Navy  F/A-18  aircrail.  The  advantages  and  deficiencies  of  previous  systems  are  summarized  below, 
followed  by  a  list  of  the  technological  features  which  were  designed  into  the  lEIMPMA  combination  in 
an  effort  to  overcome  the  deficiencies  noted. 
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TABLE  1. 


ELECTRONIC  DELIVERY  SYSTEM  TESTS 
PRIOR  TO  CURRENT  USAF/USN  F/A-18  TEST 


PERFORMING 
AGENCY _ 

Computer-based 
Maintenance  Aids 
System  (CMAS-I) 

[ref.  3] 

Air  Force  Human 
Resources 

Laboratory 

(AFHRL) 

January  1985 

APX  64  IFF  Transponder 
Set 

Navy  Technical 
Information 
Presentation  System 
(NTTPS-I)  [ref  4] 

David  Taylor 
Research  Center 
(DIRQ 

October  1986 

F-14A  Rudder  Manual 
Trim  System  (RMTS) 

CMAS  -n  [refs.  3,5] 

AFHRL/Navy 
Personnel  Research 
and  Development 
Center  (NPRDC) 

Deconber  1986 

APX  64  (V)  IFF 
lYansponder  Set 

NITPS  n  [ref  6] 

DTRC 

April  1987 

AN/SPA.25D  Radar 
Repeater  Set 

Personal  Electronic 
Aid  for  Maintenance 
(PEAM)  [ref  7] 

U.S,  Army  ARI 

March  1989 

Portable  Con^uter 
Maintenance  Aids 
System  (PCMAS) 

[ref  8] 

AFHRL 

May  1989 

F-16  Fire  Control  Radar 
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•  Ihe  optinial  association  of  text  and  gr^hics  on  the  screen  is  much  easier  for  the  technician  to 
conpehend  than  the  current  paper-tsased  formats. 

•  Technical  Manuals  in  electronic  form  require  considerably  less  maipowcr  to  i^jdate  than  their  paper 
counterparts. 

•  Software  linking  between  screens  allows  the  technician  to  gain  direct  short-term  access  to  related 
pieces  of  informatioa  The  same  access  throu^  cross-referencing  in  paper  Manuals  is  much  more 
time-consuming  and  tedious. 

•  Interactive  troubleshooting  guidance  contained  in  eleclionic  delivery  systems  is  more  effective  than 
its  counterparts  in  conventional  paper  N^uals. 


2. 1.2, 2  Deficiencies  Reported  During  Previous  Tests 

•  The  pre-established  form  of  the  troubleshooting  instructions  in  earlier  tests  restricted  the  Technicians' 
use  of  their  experience  and  knowledge  in  streamlining  a  Fault-Isolation  procedure.  Fault-Isolation 
procedures  using  feedback  fiom  previous  maintenance  actions  (e.g.,  actual  failure-rate  data)  can 
increase  the  effectiveness  of  such  procedures. 

•  Many  of  the  devices  used  to  display  the  TI,  and  especially  their  user  interfaces,  were  off-the-shelf 
commercial  items,  not  designed  for  maintenance  applications  and  less  than  optimum  in  human-factors 
design.  In  cases  Miere  special  devices  were  designed,  bulk  and  wei^t  were  problems. 

•  Most  future  and  many  current  wet^n  systems  include  Built-In  Test  (BIT)  capability,  yet  only  the 
PCMAS  device  used  in  the  F-16  test  (ref.  8)  had  a  capability  which  allowed  the  Technician  to  use 
his  display  device  to  interact  directly  with  BIT.  Such  an  interaction  provides  a  potential  for 
inpoving  the  timeliness  and  accuracy  of  diagnostics. 

•  The  maintenance  process  includes  a  substantial  amount  of  paperwork  which  lends  itself  to  computer- 
assisted  preparation  However,  previous  programs  had  not  fully  tested  this  potential  application. 

•  Graphics  were  in  general  of  poor  quality.  Display  software  was  frequently  inadequate  (e.g.,  requiring 
unacceptable  time  to  associate  the  required  text  and  graphics,  and  to  present  successive  frames  to  the 
user). 


2. 1.2.3  Technological  Advances 

As  repeated  field  tests  have  refmed  both  operational  requirements  for  lEIMs  and  an  understanding  of 
the  effectiveness  and  deficiencies  of  proposed  approaches,  the  Navy  and  the  Air  Force  have  continued 


9 


developmental  and  evaluational  efforts  in  EETM  technology.  The  IMIS  program  of  the  USAF  Armstrong 
Laboratory  (AL/HRGO)  [Refs.  8,9,10]  continued  the  efforts  started  under  the  CMAS  and  PCMAS  projects 
with  the  purpose  of  e?q)loiting  recent  technological  advances.  The  products  developed  by  applying  these 
advances  included  the  following: 

•  Portable  Maintenance  Aid  (PMA).  AL/HRQO  developed  a  portable  display  device  for  the  F/A*  1 8 
test  consisting  of  a  special-purpose  portable  conputer  specifically  designed  to  sipport  maintenance 
on  the  fli^tline.  A  key  feature  of  the  device  is  its  transflexive,  hi^-resolution,  Liquid-Crystal 
Display  (LCD)  usable  in  a  range  of  ambient  li^t  ffom  bri^t  sunli^t  to  total  darkness. 
Specifications  for  the  PMA  are  given  in  Table  2.  The  keyboard  and  screen  layout  are  shown  in 
Figure  1. 

•  Diagnostic  Module  (DM),  [ref.  10]  The  DM  is  designed  to  use  Technical  Information  extracted 
from  the  aircraft's  BIT  via  the  1553  data  bus,  and  to  integrate  this  information  with  system  Technical 
Infonnation  (e.g.,  signal-flow  information),  syn^nvfault  relationships,  test  times,  rectification  times, 
and  conponent  failure  rates.  From  these  data,  the  DM  computes  a  recommended  procedure  for 
isolating  an  observed  fault.  The  Technician  may  follow  the  recommended  procedure  or  he  may  rely 
on  his  experience  to  select  one  of  the  listed  alternative  strategies.  Also,  the  DM  has  the  capability 
of  handling  cases  involving  multiple  faults. 

•  Content  Data  Model  (CDM).  The  CDM  is  a  hierarchical  data  base  structure  similar  to  the  "topdown 
breakdown"  approach  used  to  organize  parts  and  pieces  of  a  weapon  system.  The  CDM  offers  an 
orderly  structure  for  electronically  storing  and  exchanging  software-system  independent  digital 
representations  of  the  system  data  tpon  which  the  lETM  is  constructed  From  this  data  base  (the 
lETMDB),  it  is  possible  to  extract  the  data  and  display  it  on  any  of  several  display  devices,  provided 
that  a  suitable  Presentation  System  (PS)  is  amiable  to  establish  the  proper  information  sequencing 
and  arrangement.  The  test  was  intended  to  assess  the  suitability  of  a  portion  of  the  F/A-18  data  base 
constructed  in  accordance  with  the  MIL-D-87269  revisable  data  base  Specification,  [ref.  1]  (Note 
that  this  evaluation,  involving  the  effectiveness  of  the  lETMDB  as  an  interim  measure  in  preparing 
the  lETM,  was  not  part  of  the  field  evaluation  described  in  this  report,  which  tested  only  the  final 
lETM  product.  The  lETMDB  evaluation  is  being  reported  separately  by  Armstrong  Laboratory,) 

•  Presentation  System  (PS).  The  IMIS  PS  is  a  software  package  that  extracts  data  fiom  the  lETMDB, 
organizes  it,  and  provides  the  proper  commands  so  that  it  can  be  viewed  on  specific  display  devices; 
in  this  case,  the  IMIS  PMA.  In  addition,  it  maintains  records  of  maintenance  actions  taken  and 
automatically  completes  forms  to  report  these  actions.  An  evaluation  of  the  effectiveness  of  the  PS 
in  providing  lETM  Technical  Information  is  being  performed  by  Armstrong  Laboratory. 
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TABLE  2. 


CPU 

Size 

Case 

Wei^t 

Display 

Memory 

Qraphics 

Interfaces 

Power  Siqjply 

Operating  System 
Digital  Multimeter 


IMIS  PMA  SPECIFICATIONS 
Motorola  68020-based  hybrid 
9.5"  X  10.5"  X  2.5" 

Carbon-fiber  con^site 
6  lbs 

Ovonics  6"  x  8"  monochrome  active  matrix, 
640  X  480  resolution 

6  MByte  SRAM,  4  MByte  FLASH  PROM, 
32  MByte  removable  memory  cartridge 

Intel  87871-based  hybrid 

MIUSTO  1533,  RS  232C,  1.3  MHs 
channel  hopping  radio 

16.5  Volt  Yardney  Silverccll  battery  pack 
(4  hour),  15  Volt  DC  external  output 

Application  software 

AC  Volts,  DC  Volts,  C%ms,  autoranging 
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•  Human  Computer  Interface  (HQ).  The  IMS  PMA,  as  designed  and  corifigured  in  accordance 
with  the  HCI,  permits  the  rapid  retrieval  of  desired  Technical  Infoimation,  including  information  via 
the  1553  bus  from  the  aircraft's  BIT.  Because  of  the  PMA's  small  size,  input  devices  include  only 
a  limited  keyboard  (numbers,  special-function  keys,  and  programmable  function  keys)  and  a  joystick. 
An  adaptation  of  the  Open  Software  Foundation  methodolog)',  the  HCI  uses  the  "aim  and  shoot" 
approach  for  information  access  and  manipulation  Hie  test  was  designed  to  evaluate  the  HCI  by 
fielding  an  lETM/PMA  combination  constructed  in  accordance  with  MIL-M-87268  [ref.  2],  which 
provides  requirements  for  general  content,  style,  format,  and  user-interaction  features  of  lETMs.  The 
HCI  has  been  described  in  some  detail  in  ref.  1 1. 


2.1.3  The  Interactive  Electronic  Technical  Manual  (lETM) 


The  term  lEIM  [see  ref.  12]  has  been  defined,  for  example,  b  ref.  2,  as  follows; 


A  Teohnioal  Manual,  prfparad  (authored)  by  a  Gonbraobor  and 
dallvarad  be  bha  Oovaxmant,  or  prepared  by  a  OovarzBaanb 
aoblvlby,  din  digital  form  on  a  euitablo  medium,  by  meana  of  an 
aubomabed  Authoring  Syatemi  dealgned  for  eloobronlo-aoreen 
display  bo  an  end  user,  and  posaasalng  the  folloidng  three 
oharaoterlsbles i 

a.  The  format  and  style  of  the  presented  information  an 
opbimited  for  soreen  presentation  to  assure  insx..ijniin 
ocnprehenslon)  that  la,  the  presentation  format  la 
"frame-oriented",  not  "page-orlsntad" . 

b.  Iha  elasaants  of  Teohnloal  Information  coonstl tubing  the 
IM  are  so  interrelated  that  a  uaer's  aooess  to  the 
information  ho  requires  is  faoilitabed  to  the  greatest 
extant  possible,  and  la  aohievablo  by  a  variety  of 
paths. 

o.  Iho  aonputor-aantrolled  TM-Display  device  oan  funotlon 
Intoraotively  (aa  a  result  of  uaar  requests  and 
information  ii^t)  in  providing  procedural  guidance, 
navigational  direotions,  and  supplanantal  information; 
and  also  in  providing  aasiataxios  in  oarrylng  out 
logiatlo-svvport  functions  supplanantal  to  malnteaanoo . 


Characteristics  of  such  a  presentation  have  been  described,  for  exanple,  in  ref  3,  and  general 
requirements  are  presented  in  MIL-M-87268  [ref.  2).  The  HQ  and  the  Presentation  System  of  IMIS  have 
been  designed  to  present  the  required  Technical  Information  to  a  technician  in  EBTM  form. 
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3.0  METHODOLOGY 


■ 

3.1  GENERAL  APPROACH  I 

Althou^  the  test  was  carried  out  under  normal  squadron  maintenance  conditions  to  the  extent  ■ 

possible,  some  minoi  changes  to  normal  procedures  were  necessitated  These  included: 

a.  Assignments  were  given  to  Technicians  by  test  personnel  rather  than  Maintenance  Control  and  P 

Work  Center  supervisors. 

b.  Communications  with  Si^ly  were  simulated  by  test  personnel. 

c.  One  Technician  was  asked  to  perform  all  of  the  work  of  an  action  instead  of  using  the  customary  P 

team  approach. 

d.  External  hydraulic  power  was  used  as  compared  with  the  usual  practice  of  relying  on  aircrait  ^ 

hydraulic  systems.  P 

e.  A  technically  qualified  test  svqjervisor  stayed  with  the  test  Technicians  to  assure  safety  and  to 

respond  to  any  questions  the  test  Technicians  had  I 

3.2  TEST  METHODOLOGY 

In  order  to  accomplish  the  Test  objective  (see  section  1 .3),  the  primary  methodology  of  the  Test  was  v 

to  compare  (a)  the  effectiveness  of  Technicians'  troubleshooting  performance  when  using  the  specially  m 

prepared  EETM/PMA  combination,  with  (b)  the  effectiveness  of  their  performance  when  using  paper  P 

Technical  Manuals  to  guide  their  efforts.  ^ 

Specifically,  the  following  testing  was  done; 

a.  With  each  type  of  Technical  Manual,  conpire  troubleshooting  performance  in  terms  of  time  P 

on  task,  success  in  finding  the  fault,  number  of  False  Removals,  and  Procedural  Errors.  ^ 

b.  Evaluate  user  preference  for  the  lEIMPMA  combination  with  respect  to  the  paper  TMs, 

stressing  information  access  and  presentation  effectiveness.  Solicit  user  suggestions  with  |A 
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regard  to  troubles  experienced  with,  and  proposed  improvements  for,  the  technological 
improvements  tested  (Ihese  features  are  summarized  in  section  2.1.2. 1.) 

c.  Evaluate  the  PMA  design  in  accordance  with  Hunan  Factors  principles  and  user  preferences. 

d.  Evaluate  three  principal  maintenance-sipport  capabilities  of  the  lETM/PMA  combination  on 
the  bases  of  troubleshooting  performance  measures  and  user  preferences.  These  capabilities 
were: 


(1)  The  diagnostic  interface  of  the  lETM^MA  with  the  F/A-18  Built-In-Test  Equipment 
(BITE)  throui^i  the  1553  multiplex  bus. 

(2)  The  automated  conviction  of  standard  maintenance-action  and  reporting  forms  and  parts- 
ordering  requests. 

(3)  The  integrated  display  of  diagnostic  instructions,  maintenance  procedures,  and  schematic 
diagrams  by  means  of  a  self-contained  portable  display  device  (the  PMA). 


3.3  TEST  DESIGN 

In  the  experimental  design  was  a  2x2x3-factorial  mixed  desiga  'Ihe  independent  variables  were: 

Factor  1  The  Technical  Information  Ihsplay  approaches  (two:  lETM/PMA  and  Paper  TM); 

Factor  2  Ejqxaience  levels  of  the  Technicians  (two:  Experienced  and  Inexperienced); 

Factor  3  The  types  of  faults  (three:  Relay  Faults,  Cannot  Divlicate  "faults",  and  Multiple 
Faults.  The  fault  types  are  described  in  section  3.4. 1 .).  Each  type  of  fault  had  two 
representative  faults  of  equal  conplexity;  i.e.,  six  individual  fault  situations  were 
used 

Factor  2  involved  between-grovp  conparisons;  Factors  1  and  3  involved  within-groip  comparisons. 

Each  of  the  16  Technicians  attenpted  three  Fault  Isolations  using  the  lElMPMA  combination  (one 
fiom  each  fault-type  pair)  and  three  Fault  Isolations  using  the  p^r  TMs  (the  remaining  fault  fiom  each 
pair).  From  each  e}q)erience-level  groip,  four  participants  used  the  lETMEMA  combination  first  and  four 
used  the  paper  TM  first.  This  counterbalancing  eliminated  the  effects  of  learning  bias  and  interference. 
Each  Fault-Isolation  procedure  was  performed  an  equal  number  of  times  with  lEIMPMA  and  paper-TM 


15 


support;  i.e.,  papier  TMs  were  used  for  three  Fault  Isolations  and  the  lETNFPMA  combination  was  used 
for  three  different  isolations.  To  avoid  confusion  and  interference  learning,  one  TM  type  was  used  for 
three  consecutive  Fault  Isolations,  then  the  other  system  was  used  for  the  next  three.  The  faults  assigned 
to  the  paper-TM  tests  and  the  lEIMPMA  tests  were  randomized  to  eliminate  undesirable  learning  effects 
either  positive  or  negative. 

To  accommodate  a  total  of  ei^t  participants  in  each  experience  group,  two  additional  sequences 
were  randomly  selected  in  such  a  way  that  each  Fault  Isolation  was  atterrqjted  an  equal  number  of  times 
with  each  type  of  TM,  and  that  each  Fault-Isolation  sequence  within  a  pair  was  carried  out  an  equal 
number  of  times.  As  a  last  precaution,  the  assignment  of  TM  type  to  test  Technicians  was  alternated 
between  lEIMPMA-then-px^jer  and  paper-then-IEIMPMA  The  effect  of  these  assignment  considerations 
resulted  in  the  run  schedule  shown  as  Table  3. 


3.4  FAULT  DESCRIPnONS 


3.4.1  Faults 

The  Strike  Aircraft  Test  Directorate  at  Naval  Air  Station  (NAS)  Patuxent  River  had  evaluated  a 
"Fault  Insertion  Program"  for  the  Naval  Air  Systems  Command  [refs,  13  and  14 1.  Ihc  purpose  of  the 
project  was  to  generate  and  exercise  simulated  faults  as  a  means  of  testing  the  BIT  capability  of  the  F/A- 
18  FCS.  A  total  of  188  faults  were  developed  which  could  be  "inserted"  into  the  FCS  throu^  the  use 
of  a  device  referred  to  as  a  Breakout  Box  A  total  of  34  additional  faults  were  developed  later.  Four  of 
the  faults  used  in  the  F/A- 18  lETM/PMA  test  were  drawn  from  this  pool  of 222  faults.  Three  fault  types 
were  used  as  the  basis  for  the  test;  two  faults  were  chosen  from  each  of  three  fault  types. 


16 


TABLES 


PARTICIPANT  RUN  SCHEDULE 
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2  6 

4 
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- 
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4 

6 
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3  1 
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5 
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- 
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The  three  pairs  of  faults  by  type  are: 

a.  Relay  (Electrical^  Faults  fFaults  1  and  2V 

Insertion  of  two  defective  reiays  which  had  failed  while  in  service  was  used  to  represent  these 
faults.  The  main  characteristic  of  this  fault  set  was  several  branchings  in  the  Fault-Isolation 
procedure.  The  paper  'TM  indicated  brandling  by  a  two-column  "go  to"  arrangement  which 
directed  the  Technician  to  move  to  the  next  step  or  a  step  ftirther  downstream  or  iqsstream  in  the 
step  sequence.  The  lEIMPMA  required  only  single  keying  actions  to  ii^ut  test  results  and  to 
display  the  next  action  The  two  faults  in  this  set  involved; 

1  Nose  'Wheel  Steering  Selector  Valve 

2  Left  Wing  Unlock  Relay 

b.  Can  Not  Duplicate  (CNDl  Faults  (Faults  3  and  4V 

Using  the  paper  TM  to  isolate  faults  in  this  set  required  a  Technician  to  watch  a  rapidly  scrolling 
set  of  multi-digit  numbers  in  a  window  of  his  Digital  Display  Indicator  G^DI)  and  then,  by 
referring  to  a  chart  in  a  different  location  in  the  TM,  judge  whether  the  numbers  he  had  seen  fell 
within  the  chart's  acceptable  range.  The  lETM/PMA  combination  fed  the  numbers  into  the  PMA 
(via  the  1SS3  interface  bus)  which  performed  an  initial,  automatic,  assessment  of  the  numbers. 
This  pair  of  "faults"  consisted  of  the  Can  Nd  Explicate  (CND)  faults;  i.e.,  the  correct  result  of 
the  Technicians'  task  was  to  indicate  the  absence  of  a  fault.  Thus,  no  faulty  conponents  were  in 
fact  present,  real  or  simulated  However,  the  symptoms  reported  to  the  Technicians  indicated 
faults  in  the  following  components: 

3  Right  Trailing  Edge  Flap 

4  Left  Stab  Test 

c.  Miltiple  Faults  (Faults  5  and  61 

Each  fault  of  this  fault  type  consisted  of  establishing  a  combined  set  of  symptoms  for  two  failed 
conponents.  The  insertion  of  these  faults  used  the  Breakout  Box  referred  to  earlier.  In  Fault 
Isolation  using  the  paper  'fM,  the  Technician  was  required  to  cpproach  this  Fault  Isolation  as  two 
separate  Fault-Isolation  problems,  with  the  need  to  solve  one  problem  before  moving  to  the  next. 
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with  the  IFFM/PMA  combination,  the  combined  symptom  set  was  addressed  as  a  single  fault  (the 
tests  used  to  solve  the  second  fault  [e.g.,  5  (b)]  were  a  function  of  the  tests  used  to  solve  the  first 
fault  [e.g.,  5  (a)].  The  two  fault  combinations  in  this  set  involved  defective  wiring  or  connectors 
associated  with; 


5  (a)  Trailing  Edge  Clap  and  5  (b)  Aileron  Left. 

6  (a)  Aileron  Left  Shutoff  Valve  and  6  (b)  Leading  Edge  Flap  Left. 


3.4.2  Fault  Insertion 

Inserting  the  two  Electrical  Faults  [1  and  2]  involved  replacing  good  relays  with  ones  that  had  failed 
in  service.  Hie  next  two  faults  [3  and  4]  were  CND  situations  and,  whereas  syirqitoms  were  provided,  no 
fault  was  actually  present.  Each  pair  of  Multiple  Faults  [5  (a),  5  (b)  and  6  (a),  6  (b)]  was  inserted  non- 
destructively  into  the  FCS  of  the  F/A-18  using  the  Breakout  Box. 

Hie  Breakout  Box  used  to  insert  faults  into  the  FCS  was  connected  by  cable  to  each  of  the  two  FCS 
computers  (FCCA  and  FCCB).  Each  fault  was  associated  with  one  ofthe  Breakout-Box  switches.  Labels 
on  switches  identified  the  fault,  e.g.,  in  the  label  A-Jl-2,  the  A  meant  FCC  A;  the  J1  meant  Corrputer 
Connector  1;  and  the  2  meant  pin  #2.  Hie  procedure  for  inserting  &ults  using  the  Breakout  Box  was  as 
follows: 

a.  Ensure  that  the  test  aircraft  had  been  made  safe  for  maintenance. 

b.  Apply  ground-based  electrical  and  hydraulic  power  to  the  aircraft. 

c.  Run  preflight  Initiated  BfT  (TBIT)  to  determine  that  the  aircraft  was  fault-free. 

d  Install  fault-insertion  tester  into  the  system  at  the  connectors  of  both  flight-control  con^uters 
(FCCA  and  FCCB).  Hie  Breakout  Box  was  equipped  with  jumpers  for  opening  the  connection 
between  the  FCCs  and  the  aircraft. 

e.  Run  IBIT  to  determine  that  the  FCS  had  remained  fault-free. 

f.  Set  desired  fa)ilt  on  the  Breakout  Box 

g.  Run  IBIT  to  confirm  intended  fault  state. 


19 


3.5  PREPARATION  OF  THE  irmaUCnVEiaj^CmOMCTEaiNICAI.  MANUAL  (lETM) 


3.5.1  Work  Package  Source  Data 

Selected  parts  of  the  existing  F/A-18  Work  Package  paper  TMs  were  used  as  source  data  for  the 
preparation  of  the  lETM  Technical  Infoimatioa  These  were: 

a.  Fault  Reporting  Manual  (FRN^  -  Consists  of  lists  of  BIT  fault  codes  each  of  ^\hich  refers  to 
specific  parts  of  other  WPs  for  follow-i^  troubleshooting  and  repair. 

b.  100  series  manuals  -  contain  theory  of  operation  for  the  FCS. 

c.  200  series  manuals  •  contain  troubleshooting  instructions  for  major  parts  of  the  FCS;  e.g.,  Wing 
Fold  System. 

d.  300  series  manuals  -  contain  Coirective-Maintoiance  instructions,  e.g.,  remove  and  reinstall,  align, 
rig. 

e.  500  series  manuals  -  contain  a  mixture  of  locator  drawings  and  9"  x  14"  sheets  of  schematics. 

f.  Line  Maintenance  Manual  (LMNf)  series  •  doors  and  panels. 

g.  Plane  Captain's  manual. 


3.5.2  Organizing  the  lEIM 

Material  from  the  pertinent  Work  Packages  (WPs)  of  the  Fli^t  Control  System  were  formed  into  a 
database  (lEIMDB)  organized  according  to  the  Content  Data  Model  (CDN^  concept.  Elements  of  the 
CDM  database  were  marked  with  Standard  Generalized  Markt^?  Language  (SGML)  tags  as  required  for 
data>exchange  purposes.  Before  any  part  of  this  database  could  be  viewed  on  the  display  of  the  Portable 
Maintenance  Aid  (PMA),  the  Tl  was  extracted  from  the  database  and  organized  into  a  form  that  is 
viewable  on  the  PMA  This  ordering  of  the  Technical  Information  in  the  lElMDB  was  acconplished  by 
applying  a  set  of  algorithms  (referred  to  as  the  IMIS  Presentation  System)  hosted  in  the  PMA  This 
overall  sequence  is  summarized  by  Figure  2. 
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3.5.3  lETM  Reviews 


Although,  in  general,  lEIM  Quality  Assurance  procedures  prescribed  by  MIL-Q-87270  [ref.  16], 
v^ch  had  not  as  yet  been  issued,  were  not  followed  in  preparation  of  the  Test  lETM  materials,  additional 
QA  efforts  performed  during  the  preparation  process  included; 

a.  Human  Factors  (HF)  Reviews  of  draft  lETM  to  ensure  that  the  information  as  shown  on  the 
lEIMPMA  display  did  not  deviate  from  accepted  HF  criteria,  as  reflected  in  the  Human 
Computer  Interface  (HCI)  [ref  11],  in  MIL^M-87268  (ref  2),  and  other  relevant  HF  literature, 
In  reviewing  the  draft  lETM  H,  HF  personnel  also  took  into  account  the  following  considerations: 

(1)  Adequacy  of  amount  of  material.  (Is  the  amount  of  text  and  graphic  information  usable  by 
the  Technician?). 

(2)  Ease  of  following  progressive  sequences  of  illustrations  (e.g.,  fiem  work  area  to  panel  to 
relevant  part). 

(3)  Effectiveness  of  callouts  (the  number  of  callouts  and  the  order  of  their  appearance). 

(4)  Legibility  of  relevant  graphic  items.  (At  noimal  reading  distances,  the  items  should  be  large 
enough  to  subtend  a  12*minute  arc  at  the  viewer's  eye). 

(5)  Adequacy  of  the  user/PMA  interface,  iircluding  the  following  considerations; 

(a)  All  required  functions  included? 

(b)  Placement  of  priority  keys. 

(c)  Means  of  cursor  control,  e.g.,  joystick  or  cursor  arrow  keys. 

(d)  Ease  of  operation,  e.g.,  retrieval,  navigation  among  different  types  of  Tl. 

b.  Performance  of  lETM  Validation  and  Veriflcatioa 
(1)  Validation. 

In  preparing  new  Technical  Information,  either  in  paper  form  or  as  an  lETM,  Validation  is 
a  Contractor  function  which  must  be  performed  on  the  weapon  system  (aircraft  or  aircraft 
con^nent)  involved,  using  actual  TI  which  is  displayed  on  the  same  Display  Device  that  is 
to  be  used  by  the  maintenance  techniciaa  The  purpose  of  Validation  is  to  guarantee  that  all 
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TI  content  necessa»y  to  perform  maintenance  is  present  and  accurate.  For  purposes  of  the 
F/A-18  test,  however,  Validation  by  MCAIR  consisted  of  the  following: 

(a)  Determined  that  the  textual  contents  of  the  paper-TM  and  lETM  Technical  Information 
forms  were  in  conformance. 

(b)  Determined  that  the  locator  gre^hics  in  paper  form  were  equivalent  to  (presented  the 
same  information  as)  the  locator  graphics  in  lETM  form. 

(c)  Determined  that  the  text  and  graphics  in  the  lETM  were  linked  correctly. 

(d)  Determined  that  all  lETM  branching  was  present  and  that  there  were  no  dead  ends. 

(e)  Determined  that  the  lETM  was  viewable  and  usable  on  the  IMIS  PMA. 

(2)  Verification  is  a  Government  responsibility  performed  on  the  aircraft,  using  military 
technicians  and  lEIMPMA.  Ihe  purpose  of  Verification  is  to  assure  the  usability  of  the 
lETM/PMA  in  a  realistic  environment.  A  formal  Verification  was  not  conducted.  Instead, 
during  the  Pretest  which  preceded  the  test,  the  test  team,  including  operational  personnel, 
reviewad  the  lEIMPMA  to  assess  tlw  usability  of  the  lETM.  Problems  were  corrected  prior 
to  the  actual  test. 


3.6  DESaUPTION  OF  SUBJECT  TEa^MaA^^ 

Sbcteen  Marine  Corps  Technicians  participated  in  the  test.  All  possessed  specialties  of  either 
Electrician  (Military  Occi^wtional  Specialty  6337)  or  Communications/Navigation/  Radar  (Military 
Occupational  Specialty  6317).  Ei^t  of  the  Technicians  were  judged  by  their  supervisors  to  be 
experienced  in  F/A-18  Flight  Control  System  (FCS)  maintenance;  each  of  these  Technicians  had  at  least 
24  months'  experience  of  hands-on  work  on  the  FCS.  The  remaining  ei^t  Technicians  were  Judged  to 
be  inejqrerienced,  having  18  months  or  less  on  the  FCS,  with  the  average  at  10.25  months.  Fifteen  of  the 
sbcteen  Technicians  were  male;  one  was  female. 

Participation  in  the  test  was  voluntary.  All  participants  were  briefed  on  the  general  nature  of  the  test. 
All  signed  an  Informed  Consent  Form. 
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3.7  DESCRIPTION  OF  THE  TEST  FAaUTY 


The  Test  Facility  included  two  external  hjdraulic  generators,  the  Breakout  Box,  a  workstand  for 
cockpit  entry  and  exit,  an  intercom  system,  Workcenter  spaces  with  TMs,  briefmg/debriefmg  areas  and 
a  portion  of  the  hangar  for  the  aircraft  and  its  seaport  equipment.  Technicians  were  siqjplied  with 
appropriate  tool  boxes  and  other  required  equipment,  The  same  hangar  space  was  used  throu^out  the 
test. 


3.7.1  The  FCS  and  the  Aircraft 

The  test  vehicle  was  the  Flight  Control  System  (FCS)  installed  in  a  single  F/A-18  aircraft  (used 
throu^out  tlte  test):  Model  C,  Lot  XE.  The  aircraft  was  i^ovided  by  VMFA-3 12,  MCAS,  Beaufort,  SC, 
Parked  inside  a  hangar  with  all  (ground-based)  power  connected,  the  aircraft  and  its  support  equipment 
were  roped  off  for  safety  purposes.  All  aircraft  doors  required  for  maintenance  access  remained  open. 


3.7,2  External  Hydraulic  Generators 

Standard  diesel-powered  hydraulic  generators  were  used  to  provide  hydraulic  power  during  execution 
of  the  test  problems.  VMFA-3 12  provided  a  qualified  operator  to  run  the  generators.  Refueling  and 
monitoring  the  condition  of  the  generators  (especially  by  monitoring  the  ten^wrature  and  by  running 
occasional  checb)  was  shared  by  VMFA's  Safety  Monitor  and  the  generator  Operator. 


3.7.3  Breakout  Box 

The  Breakout  Box  (see  section  3.4.1)  is  described  in  ref.  13. 


3.7.4  Workstand 

A  workstand  of  the  type  tiiat  rolls  to  the  side  of  the  aircraft  was  used  to  facilitate  personnel 
movement  between  the  coclq)it  and  the  hangar  floor,  and  to  provide  a  perch  for  the  team's  technical 
observer  while  the  Technician  was  performing  cockpit  checks. 
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3.7.5  !\pin 


An  intercom  was  used  to  provide  communication  among  the  Technician  performing  the  maintenance 
task,  the  team's  technical  observer,  and  the  data  collector.  The  most  frequent  intercom  communications 
were  betsveen  the  technical  ob£;erver  and  the  test  Technician  and  between  the  technical  observer  and  the 
data  collector,  e.g.,  the  teclinical  observer  informed  the  data  collector  of  the  "handshake"  (successful 
connection)  between  the  15S3  multiplex  bus  and  the  PMA,  or  conviction  of  a  cockpit  check,  neither  of 
which  is  observable  fix>m  the  hangar  floor,  t}%  data  collector's  post. 


3.7.6  Workcenter  Spaces 

Thiee  separate  work  areas  were  provided  in  the  VMFA-312  hangar  and  Administration  Building. 
Space  for  the  parked  aircraft  and  its  sivport  equipment  was  provided  directly  outside  of  the  Workcenter 
office  and  work-cage  areas.  One  work-cage  space  (in  the  hangar  building)  was  provided  for  test  team 
equipment  and  personnel,  and  for  Technician  debriefmg  following  each  problem.  Conference-room  space 
was  provided  in  the  Administration  Building  for  Qientation  Briefmgs,  form  convletion,  and  PMA/WP 
training, 


3.8  DATA  CX)LLECnON 


3.8.1  Process 

The  following  paragraphs  identify  the  types  of  data  collected  and  the  means  used  for  data  collection. 

a.  PMATiaining.  The  PMA  training  session  consisted  oforal  descriptions,  demonstrations  of  PMA 
capabilities,  and  hands-on  Technician  tryout  and  practice  of  these  capabilities.  Frequently, 
Technicians  made  favorable  or  critical  comments  regarding  various  features  of  the  device  or  its 
displayed  Technical  Information  during  these  sessions.  The  trainers  documented  these  comments 
so  that  they  could  be  included  in  the  Debriefmg  Comment  analysis. 

b.  Biographic  Data  Sheets.  Ttiesc  forms  asked  for  background  on  each  Technician,  e.g.,  service  time, 
technical  schools  attended,  time  ui  squadron,  and  amount  of  hands-on  e^qserience.  This 
information  was  used  to  establish  the  differences  between  personnel  assigned  to  each  experience 
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level;  and  if  any  Technician  performed  unusually  well  or  poorly,  to  disclose  whether  or  not 
anything  in  his  background  could  be  the  cause  of  this  extraordinary  performance.  An  example 
of  this  form  is  provided  in  Appendix  B. 

c.  Performance-Observation  Forms,  These  forms  (a  sample  is  provided  in  Appendix  C)  listed  four 
standard  "Intervals"  (see  section  2.2.2)  for  each  Fault-Isolation  task.  (The  CND  problem  type  did 
not  include  the  Parts-Ordering  Interval.)  The  data  collectors,  in  close  communication  with  the 
technical  observer,  entered  start  times  for  each  Interval  (the  stop  time  for  an  Interval  was  the  start 
time  for  the  following  Interval).  The  form  provided  space  for  entering  notes  on  observed  errors. 
Errors  tended  to  be  of  three  types;  (1)  Procedural  (took  the  wrong  branch  after  a  test  outcome), 

(2)  False  Rentovals  or  (3)  Failure  to  Isolate  die  Fault.  The  data  collectors  noted  any  other 
significant  behavior  (good  and  bad)  for  discussion  with  the  Technician  during  Debriefmg. 
Technicians'  peiformance  times  and  procedural-error  data  were  analyzed  statistically;  content 
analyses  were  performed  on  the  Debriefmg  comments. 

Time  and  enor  data  of  the  type  discussed  above  were  collected  for  each  of  the  Ibllowing 
sequential  test  Intervals  (see  section  3.8.4); 

(1)  Preparatioa 

(2)  Fault  Isolation 

(3)  Parts  Ordering. 

(4)  Maintenance  Close-out. 

d.  E>ebriefing.  Notes  on  any  significant  behavior  wftich  the  data  collectors  observed  during  a 
Technician's  performance  were  entered  on  the  Performance  Observation  Form.  Later  during 
debriefmg,  the  data  collectors  used  these  notes  to  initiate  Debriefmg  discussions,  which  were  held 
after  each  problem  performance.  Two  sample  videos  were  made  and  retained  to  illustrate  the 
nature  of  these  interviews;  one  for  an  eiqierienced  Technician  and  one  for  an  ineiqierienced 
Techniciaa 

e.  User-Evaluation  Questionnaire.  After  fuiishing  the  six  Fault-Isolation  tasks,  each  Technician  was 
asked  to  complete  a  User-Evaluation  Questionnaire  (a  form  is  provided  in  i^ipendix  D).  Thus, 
each  Technician's  bases  for  responding  to  the  Questionnaire  items  consisted  of  three  maintenance 
tasks  guided  by  the  lETMPMA  combination  and  three  guided  by  paper  TMs.  (See  Table  3). 
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Each  use  of  the  lETM/PMA  in  a  particular  Fault-Isolation  effort  was  paired  with  a  papcr-TM  use 
in  support  of  tlic  otlier  test  of  that  type.  A  Technician  responded  to  items  of  the  Questionnaire 
by  selecting  one  of  five  scale  values  indicating  the  Technician's  opinions  regarding  the  pertinent 
lETM  or  PMA  feature.  The  scale  values  ranged  from  Unsatisfactory  to  Outstanding.  The 
Questionnaire  items  covered  the  lETM/PMA  areas  of: 

(1)  PMA  physical  dimensions. 

(2)  Keyboard  layout. 

(3)  Display  characteristics  (brigjhtness,  contrast). 

(4)  Conprehensibility  of  lETM  formats. 

(5)  Cursor,  menu,  and  information-access  features. 

(6)  Text  and  graphics  adequacy. 

(7)  Automatic  form-conpletion  procedures. 

(8)  Other  comments. 

f.  Structured  Interview.  The  structured  interview  posed  16  questions  to  elicit  the  Technicians' 
opinions  of  the  PMA  A  mqjor  purpose  of  the  interviewer  was  to  use  these  questions  as  the  basis 
for  e^qsanding  the  discussion  into  other  areas.  Two  of  the  sbeteen  questions  dealt  with  "most  liked 
-  least  liked"  lETM/PMA  features.  A  copy  of  the  Structured  Interview  Guide  is  provided  in 
i^pendbcE. 

g.  Team  Personnel  Observation  The  final  source  of  data  was  the  interaction  of  the  test-team 
members  with,  and  observation  of,  the  participating  Technicians.  Each  team  member  documented 
his  observations  on  the  design  and  use  of  the  lETM/PMA  combination  Comments  included  such 
opinions  and  such  factors  as  consistency  with  the  way  the  squadron  does  maintenance,  ways  that 
lEIMPMA  usage  mi^t  be  e>q3anded,  and  the  features  of  the  device  itself.  Comments  which 
were  not  already  covered  by  other  sources  became  a  part  of  the  Debriefing  Comments  which 
appear  in  section  4.2  of  this  report. 
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3.8.2  Test-Team  Personnel 


The  test-team  personnel  who  administered  the  F/A-18  lEIMPMA  field  test  were  as  follows: 

a.  Test  Director  (One  per  shift).  Two  scientists,  one  fitjm  AL/HRGO  and  one  fiom  NCC&OSC, 
were  responsible  for  directing  the  overall  effort.  Specific  responsibilities  of  tlie  Test  Directors 
included; 

(1)  Acting  as  the  Qovemment's  agents  in  matters  involving  planning,  scheduling  and  conduct  of 
the  test. 

(2)  Providing  principal  liaison  among  NAVAIR,  VMFA-312,  CDNSWC  and  AL/HRGO. 

(3)  Ensuring  smooth  operation  of  the  pilot  study  and  the  test,  as  well  as  proper  evaluation  of  their 
results. 

b.  Naw  Liaison  (One,  from  CDNSWC).  This  test  was  co-sponsored,  by  the  Navy  (NAVAIR  41 1) 
and  the  US  Air  Force  (AL/HRGO).  Liaison  was  provided  between  the  test  team  and  the 
sponsoring  organization  throughout  the  development  of  the  lEIMPMA  combination  and  the  Test 
Plan  (ref,  IS);  and  the  conduct  of  the  Pretest  OHlot  Test)  and  the  Test  including  the  data  analysis 
and  report  preparatioa  Specific  duties  included: 

(1)  Progress  reporting  and  resolution  of  problems  during  lETMPMA  developnvent,  Test  Plan 
development,  test  conduct,  and  report  preparatioa 

(2)  Site  selection  for  the  lETM/PMA  tryouts,  pcrfonnuncc  of  Uic  Piclust  and  Test,  especially 
throu^  interaction  with  VMFA-312  and  COMNAVAIRLANT. 

c.  Safety  Monitor  (One  per  shift).  Safety  monitors,  provided  by  VMFA-312.  were  senior 
Technicians  qualified  in  all  aspects  of  the  F/A-18  FCS.  Primary  responsibilities  were  to  monitor 
Technicians'  performance  in  order  to: 

(1)  Ensure  safety,  especially  for  the  Technicians  serving  as  participants. 

(2)  Resolve  conflicts  concerning  the  use  of  VMFA-3 12  resources,  e.g.,  aircraft,  test  equipment, 
stsaces,  personnel,  and  Technical  Infoimatioa 

(3)  Codrdinate  the  scheduling  and  s^jport  of  Technicians  serving  as  test  participants. 
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(4)  Ensure  that  no  damage  was  done  to  any  of  the  equipment  involved  (the  aircraft  PCS,  test 
equipment,  and  siqsport  equipment). 

(5)  Assist  test  participants  in  the  performance  of  multi-person  tasks  (in  accordance  with  an 
established  test-performance  protocol). 

d.  Technical  Observers  (One  per  shift  fix>m  AL/HRGO  Contractors  MCAIR  and  SEI).  These 
Contractor  personnel  observed  the  participating  Technicians'  step-by-step  actions  throu^ut 
performance  of  the  test  tasks.  These  personnel  were  erqsert  on  the  F/A- 1 8  PCS,  its  maintenance, 
the  paper-based  TMs,  and  the  lEIMPMA  combination.  Their  responsibilities  included: 

(1)  Safeguarding  personnel  and  equipment. 

(2)  Communicating  performance  times,  errors,  and  task  status  to  the  Data  Collectors. 

(3)  Evaluating  Procedural  Errors  and  False  Removals;  and  judging  success  or  failure  in  Fault 
Isolation. 

e.  TVainers.  (Three,  scheduled  as  necessary  to  meet  training  needs.  Training  personnel  consisted  of 
Air  Force  officers  on  the  staff  of  AITHRGO  and  a  representative  of  the  University  of  Dayton 
Crew  Systems  Ergonomics  Informatics  Analysis  Center).  These  personnel  developed  and 
administered  the  lEIMPMA  and  paper-TM  training  programs,  both  initial  and  refitsher  sessions. 
Responsibilities  iiKluded: 

( 1 )  Administering  the  Overview  Briefmg,  the  Informed  Consent  Form  and  the  Biographical  Data 
Forms, 

(2)  Interacting  with  the  Test  Directors  to  establish  iqxoming  run  schedules  and  associated  training 
needs. 

(3)  Administering  initial  training,  criterion  testing,  atKl  remedial  training  as  necessary;  and 
providing  refresher  training  for  both  the  lEIMPMA  and  the  paper  TMs. 

f.  Data  Collectors  (Two  per  shift,  consisting  of  civilian  and  military  staff  members  of  AI7HRGD 
and  a  representative  of  Scientific  Management  Associates).  Data  collectors  recorded  Technicians' 
performance  times  and  error  data,  and  solicited  opinion  data  during  debriefing  sessions.  Data 
collector  efforts  ovcrliqjped;  that  is,  while  Collector  Number  1  monitored  the  performance  of  one 
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Technician,  Collector  Number  2  was  debriefing  the  previous  Technician.  Specific  responsibilities 
of  this  position  included: 

(1)  Assuring  a  smooth  flow  of  test  participants. 

(2)  Nfonitoring  the  performance  of  the  test  participants  and  recording  observations  on  the 
Performance  Observation  Forms. 

(3)  Participating  in  the  resolution  of  problems,  misunderstandings,  or  procedural  issues  with  the 
test  Technicians  and  other  test-team  members. 

(4)  Administering  the  User  Questionnaire  and  Structured  Interview  following  a  test  participant's 
performance. 

g.  Technical  Support  (Three  personnel  scheduled  as  necessary  throughout  the  testV  Ihese  personnel 
were  PMA  software  and  Itardware  e?q)erts.  The  positions  were  filled  by  personnel  under  contract 
to  AL/HRGO.  Specific  duties  included: 

(1)  Monitoring  the  status  of  the  PMAs,  drives,  and  batteries  including  the  use  and  recharge 
cycles. 

(2)  Monitoring  the  operation  of  the  PMA  hardware  and  software;  resolving  any  problems 
encountered  with  the  device  during  the  test. 

(3)  Providing  video  recordings  during  the  structured  interviews. 

3.8.3  Materiab  and  Recording  Forms 

All  materials  and  forms  used  in  the  test  are  listed  below.  Their  use  has  been  described  in 
section  3.8.1.  Sandies  of  four  of  these  materials  appear  as  Appendbces  B  throu^  E. 

1.  Orientation  Briefing  5.  Performance  Observation  Form  (C) 

2.  Informed  Consent  Form  6.  User  Evaluation  Questionnaire  (D) 

3.  Biographic  Data  Sheet  (B)  7.  Structured  Interview  Form  (E) 

4.  Training  Syllabus 

lETM'PMA 
Paper  TMs 
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3.8.4  Performance  Measures  and  Quantitative  Data 

Data  cited  in  the  following  paiagn^^  were  collected  throu^ut  the  test.  All  maintenance  actions 
were  divided  into  seven  sequential  Intervals  to  accommodate  collection  of  performance  times. 


3.8.4. 1  Definitions  of  Test  Intervals 

a.  Preparation  Interval.  Started  when  subject  (while  in  the  Work  Center)  was  handed  the 
VIDS/MAF  describing  the  synptom;  it  ended  when  he  arrived  at  the  aircraft  with  his  collected 
resources. 

b.  FaulMsolation  Interval.  Started  when  the  Technician  arrived  at  the  aircraft;  it  et^ded  either  when 
he  announced  the  correct  cause  of  the  fault  syn^om  (i.e.,  identified  the  faulty  WRA)  or  when 
the  pre-established  maximum  time  limit  was  reached. 

c.  Wire-Repair  or  WRA-Removal  Interval.  WRA  Removal  started  when  the  Technician 
announced  the  conect  cause  of  the  disoepency;  it  ended  when  the  removal  procedure  had  been 
found  and  veiy  briefly  reviewed  with  the  Technical  Observer.  (The  Technical  Observer  told  the 
Technician  to  assume  that  the  repair  had  been  done). 

Wire  Repair  started  when  the  Technician  announced  the  cause  of  the  dlscreixmcy;  it  ended 
when  the  wire-repair  procedure  had  been  found  and  very  briefly  reviewed  with  the  Technical 
Observer.  (The  Teclinical  Observer  told  the  Technician  to  assume  that  the  repair  had  been  done). 

d.  Parts-Ordering  Interval.  Started  when  die  Repeir/Removal  review  was  complete;  it  ended  when 
the  appropriate  portions  of  the  VIDS/MAF  were  conplete. 

e.  Reinstallation  Intervals  started  when  the  Parts-Ordering  portion  of  the  VIDS/MAF  was  con^lete; 
it  ended  when  the  reinstallation  {xocedure  had  been  located  and  reviewed  very  briefly  with  the 
Technical  Observer.  (As  with  Interval  c.,  the  procedure  was  assumed.) 

f.  System  Ibalth-Check  Interval.  This  Interval  was  carried  out  to  determine  the  fault  status  of  die 
aircraft;  i.e.,  fault-fiee  if  the  Fault-Isolation  procedure  was  correct.  It  was  usually  a  shortcut  chock 
acconplished  by  the  Technical  Obso^er.  It  started  when  the  review  of  the  Reinstallation  or 
Repair  was  complete;  it  enJod  when  the  results  of  the  Check  were  announced  by  the  Technical 
Observer. 

g.  Mdntenance  Gose>  Out  Interval  started  when  the  positive  (fault-fiee)  Health-Check  results  were 
announced;  it  ended  when  the  Technician  had  conpleted  all  entries  in  the  VIDS/MAF. 
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3.8.4.2  Data  Collected 


For  completeness,  performance-time  data  were  collected  for  all  seven  Test  Intervals,  but  due  to  the 
simulated  nature  of  parts  of  the  Wire-Repair  or  WRA-Removal  Interval,  the  Reinstallation  Interval,  and 
System-Health  Check  Interval  (necessary  to  assure  continuity  of  the  entire  maintenance  process),  only  the 
data  from  the  Preparation,  Fault-Isolation,  Parts-Ordering,  and  Maintenance  Close-out  Intervals  are 
reported  and  analyzed  in  this  report.  Specifically,  the  follotving  data  are  presented  and  analyzed: 

a.  Preparation-Time  Interval  -  Ihe  elapsed  time  from  the  initial  receipt  of  the  VIDS/MAF  to  the 
time  the  Technician  arrived  at  the  aircraft  with  all  tools  and  resources. 

b.  Fault-Isolation  Interval  -  The  time  required  to  perform  diagnostic  and  testing  actions,  e.g.,  time 
from  arrival  at  the  aircraft  to  the  time  when  faulty  WRA  was  identified  (or  the  time  limit  was 
reached  without  selection). 

c.  Parts-Ordering  Interval  -  Time  required  to  fill  out  the  parts-requisition  form  This  Interval 
started  immediately  ipon  identifying  the  faulty  component  and  ended  when  the  parts  requisition 
form  had  been  conpleted.  (Data  of  this  type  were  collected  only  for  the  Relay  Tests.) 

d.  Maintenance  Oosc-Out  Interval  -  Hie  elapsed  time  to  obtain  and  enter  the  data  required  for  the 
VIDS/MAF. 

e.  Overall  Time  to  Maintenance  Completion  •  The  total  time  required  to  complete  the  processing, 
consisting  of  all  four  actions  (cited  in  a.  throu^  dL,  above). 

f.  Failure  to  Identify  the  Fault  (FI)  -  Techrucian's  failure  to  identify  the  failed  component. 

g.  False  Removal  (FR)  -  An  incident  in  which  the  Technician  recommended  the  removal  of  a  non- 
faulty  component  erroneously  believing  it  to  be  the  cause  of  the  fault  symptom. 

h.  Procedural  Errors  -  The  number  of  procedural  errors,  such  as  misinterpretation  of  the  H  or 
inproper  use  of  the  test  equipment. 

i.  Total  Errors  -  The  total  number  of  Failures  to  Identify  Fault,  False  Removals,  and  Procedural 
Errors. 


3.8.5  Descriptive  Measures 

Analyses  were  performed  on  data  collected  from  the  Biognphic  Data  Sheet,  User  Evaluation 
Qjpstionnaire,  and  Structured  Interview. 
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The  biogn^hic  data  detemtined  the  extent  to  which  the  Technician  participants  assigned  to  the  tw/o 
levels  of  experience  differed  on  such  variables  as: 

a.  Current  enlisted  pay  grade. 

b.  Years  and  months  of  aircraft  maintenance  oqserience  in  general. 

c.  Years  and  months  of  aircraft  maintenance  eqserience  on  the  PCS. 

d  Nuniber  of  enlisted  occi^ional  specialties  held 

e.  Previous  computer  usage. 

These  analyses  indicated  that  the  two  groips  were  significantly  different  in  levels  of  maintenance 
ejqjerience. 

Responses  to  the  first  three  sections  of  the  User  Evaluation  Questionnaire  were  analyzed  both 
numerically  and  qualitatively. 

In  the  numerical  analysis,  mean  ratings  were  conpited  for  items  related  to: 

a  Physical  features  of  the  IMS  PMA. 

b.  Its  operation  and  software  features. 

c.  The  relative  efficiency  arid  effectiveness  of  the  IEIMcoiTq>ared  to  p^rnanuals.  Further,  the 
open-ended  comments  provided  in  the  final  section  of  the  questionnaire  were  examined  as  possible 
e?q}lanations  for  uniformly  low  ratings,  and  to  identify  suggestions  the  test  participants  had  for 
inpoving  the  PMA  used  in  the  test. 

Content  analyses  of  test  participants'  responses  to  the  structured-interview  items  concentrated  on 
identifying  and  categorizing  those  aspects  of  the  lETMPMA  operations  and  usage  that  may  require 
corrective  action  or  further  RDT&E.  In  addition,  analysts  noted  those  features  of  tlie  lElMPMA  that 
were  particularly  liked  by  the  test  participants  and  sipilemented  them  with  impressions  of  the  test  team 
members  where  appropriate. 
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4.0  RESULTS 


This  Section  describes  the  findings  of  the  field  test  of  the  F/A-18  lEIMPMA  combinatioa  The  data 
include; 

a  The  quality  of  Technician  performance  while  using  the  lETM/FMA  as  compared  with  performance 
using  conventional  paper  TMs. 

b.  Technicians'  comments  gathered  during  the  various  debriefings  and  interviews. 

A  discussion  (section  5)  follows  the  presentation  of  each  of  these  sets  of  findings. 


4.1  PERFORMANCE  RESULTS 

This  section  presents  the  performanceefFectiveness  findings  including; 

a  Performance  times  (Preparation,  Fault-Isolation,  Parts-Ordering,  Maintenance  Close-Out  Intervals; 
and  Overall  Total). 

b.  Errors  (Procedural  Errors,  Failures  to  Identify  Fault,  and  False  Removals). 

4.1.1  PerfomuDce  Tillies 


4.1. 1.1  Pfirfonnancc  Tiin6a,.fQr..fiifi,Eapaatioa.lntfirYal 

The  Preparation  Interval  started  when  the  data  collector  handed  the  VIDS/MAF  in  either  peqier  or 
lETM  form  (i.e.,  displayed  on  a  PMA)  to  the  Technician;  the  Interval  ended  when  the  Technician  (having 
assembled  all  tools,  TI,  and  materials  needed  for  the  test  at  hand)  arnved  at  the  aircraft.  Table  4  shows 
the  mean  Preparation  times  by  experience  level,  by  fault  type,  and  by  presentation  medium.  Figures  3 
and  4  show  performance  times  for  this  Interval,  with  comparisons  based,  respectively,  on  the  two 
experience  levels  of  the  Technicians,  and  on  the  two  types  of  medium.  For  tests  in  which  lElMs  were 
used,  the  Technical  Information  required  had  already  been  ftilly  loaded  into  the  PMA  (by  the  Work-Center 
staff,  prior  to,  or  as  part  of,  the  assignment  ofthe  Technician  to  the  Fault-Isolation  task),  but  for  tests 
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NOTE:  TABLE  4  presents  al  data  oUamed  for  the  Preparation  Interval 
(see  paragrc^ih  4.1.1 .1}  for  each  of  the  96  tests  performed.  (See  TABLE  3.) 


FIGURE  3 

COMPARISON  OF  PERFORMANCE  OF  INEXPERIENCED  TECHNICIANS 
WITH  THAT  OF  EXPERIENCED  TECHNICIANS. 

(AVERAGE  PERFORMANCE  TIMES  IN  MINUTES  REQUIRED;  ALL  TESTS) 
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FIGURE  4 

Comparison  of  Performance  of  Technicians  Using  lETMs  with  that  of 
Technicians  Using  Paper  TMs,  for  Preparation  Intervai 
(Average  Times  in  Minutes  Required;  Aii  Tests) 

RELAY  TEST 


using  paper  TMs,  the  time  required  to  locate  the  appropriate  paper-based  11  ^vas  included  in  the 
performance-time  record.  Thus,  the  Preparation-Interval  measurement  was  designed  to  assess  Technicians' 
time  required  to  assemble  paper-based  TI  for  an  assigned  task:  [Time(paper)  -  Time(IETM)]  =>  Time  to 
assemble  paper  TI,  wWch  is  the  primary  time-consuming  effort  in  this  short  Interval. 

Table  S  shows  the  average  ratios  of  Technician  performance  times  for  the  Preparation  Interval  with 
lETM  use  to  those  with  paper-TM  use.  Note  that  in  every  case  this  ratio  is  less  than  1;  i.e.,  for  this 
Interval,  use  of  lETMs  decreased  performance  times  for  all  combinations,  by  the  factors  shown. 


4. 1,1. 2  Performance  Times  for  Fault  Isolation 

The  Fault-Isolation  Interval  started  when  the  Technician  arrived  at  the  aircraft  and  ended  when  the 
Technician  announced  his  finding;  e.g.,  "This  is  a  CND".  Table  6  presents  mean  performance  times  for 
this  Interval  by  esqierience  level,  by  fault  type,  and  by  type  of  TM  used.  Figures  5  and  6  show 
performance  times  for  this  Interval  with  comparisons  based,  respectively,  on  the  two  eNperience  levels  of 
the  Technicians  and  on  the  two  types  of  medium.  Table  7  shows  the  ratios  of  perfomiance  times  for  the 
Fault-Isolation  Interval  with  lEIM  use  to  those  with  paper-TM  use. 


4. 1.1. 3  Performance  Times  for  Parts  Ordering 

Hie  Parts-Ordering  Interval  commenced  when  the  Wire-Repair  or  WRA  Removal  Interval  had  been 
convicted  (see  section  3.8.4)  and  ended  with  the  ccrrqjletion  of  the  Parts-Ordering  form  (Parts  Ordering 
was  not  performed  for  the  CND  tests).  Table  8  presents  the  mean  performance  times  for  the  Parts- 
Ordering  Interval  by  fault,  by  eTqxrience  level,  and  by  medium  Figures  7  and  8  show  performance  times 
for  this  Interval  with  conparisons  based,  respectively,  on  the  two  ejqserience  levels  of  the  Technicians  and 
on  the  two  types  of  medium  Table  9  shows  the  ratios  of  performance  times  for  the  Parts-Ordering 
Interval  with  lETM  use  to  those  with  paper-TM  use. 
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TABLE  6 

Effect  of  Using  lETMs  Instead  of  Paper  TMa 
in  Preparation  Interval  (All  Testa) 


AVERAGES  OF  INDIVIDUAL  RATIOS  OF 

PERFORMANCE  TIME  (lETM) 
PERFORMANCE  TIME  (PAPER) 

for  each  technician 


RELAY  TESTS 

Experienced  Technicians 

Inexperienced  Technicians 

IHHEiSiHlii 

CND  TESTS 

Experienced  Technicians 

Inexperienced  Technicians 

oSoi 

0.477 

MULTIPLE  TESTS 

Experienced  Technicians 

Inexperienced  Technicians 

0.480 

0.436 

riME(imfUTES)  REC2UIRED  FOR  PERFORMANCE  OF  FAULT  ISOLATION 
(FAULT-ISOLATION  INTERVAL;  ALL  TESTS) 
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FiaURB  5 

Compariaon  of  Performance  of  Experienced  Technlclana  with  that  of 
Inexperienced  Technicians,  for  Fault-Isolation  Interval 
(Average  TImea  in  Minutes;  All  Tests) 


riOURB  6 

Comparison  of  Perfomiancc  of  Technicians  Using  lETMs  with  that  of 
Technicians  Using  Paper  TMs,  for  Fault-Isolation  Interval 
(Average  Times  in  Minutes;  All  Tests) 


I 

I 


TABLE? 

Effect  of  Using  lETMs  Instead  of  Paper  TMs 
In  Fault-Isolation  Interval  (All  Tests) 


AVERAGES  OF  INDIVIDUAL  RATIOS  OF 

_ PERFORMANCE  TIME  (lETM) 

PERFORMANCE  TIME  (PAPER) 

for  eaoh  technician  ■  8  n 

1  ?"|*(ietm)I 

n  8 1  t(PAPER)l 


RELAY  TESTS  | 

Experienced  Technicians 

Inexperienced  Technicians 

CND  TESTS 

Experienced  Technicians 

Inexperienced  Technicians 

0.633 

0.880 

MULTIPLE  TESTS  | 

Experienced  Technicians 

Inexperienced  Technicians 

HHEES9HHi 

Note  that  the  above  ratio  is  not,  in  general. 

equal  to; 


(lETM)l 

IT  8 1 


8  n 

J_  ^(PAPER)I 
n  8 1 
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TABLE  8 


TIMES  (MINUTES)  REQUIRED  FOR  PERFORMANCE  OP  PARTS  ORDERING 
(PARTS-ORDERINQ  INTERVAL;  NOT  APPLICABLE  TO  CND  TESTS) 


TECHNICIANS 

RELAY 

MULTIPLE  I 

EXRERIENgEP 

lEIM 

Bafiftc 

Ratio 

lEBd 

Paper 

Ratio 

S1 

B 

23 

0,217 

12 

21 

0,571 

S2 

11 

24 

0.458 

9 

9 

1.000 

S3 

12 

13 

0.923 

7 

17 

0.412 

S4 

7 

6 

1.167 

12 

33 

0.364 

S5 

6 

3 

2.000 

8 

22 

0.364 

S6 

6 

9 

0.667 

9 

27 

0.333 

S7 

4 

7 

0.671 

10 

26 

0.386 

S12 

6 

8.6 

0.688 

8 

29.5 

0.271 

AVERAGE: 

— 11 

11.69 

0.824 

9.38 

23.06 

0.462 

lETM 

Paper 

Ratio 

lETM 

PiPJC 

Ratio 

89 

6 

34 

0.147 

16 

26 

0,615 

S10 

7 

16 

0.438 

14 

29 

0.483 

S11 

10 

5 

2.000 

13 

20 

0,660 

S13 

13 

12 

1.083 

10 

14 

0.714 

S14 

6 

11 

0.545 

18 

17 

1.059 

S16 

6 

12 

0,867 

10 

22.6 

0.444 

S17 

10 

22 

0.465 

10 

36 

0,278 

S1B 

16 

20 

0.760 

18 

36 

0,467 

AVERAGE: 

9.26 

16.60 

0.761 

13.38 

24.94 

0.6BB 
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MINUTES 


FIOURB  7 

Comparison  of  Peribrmance  of  Experienced  Technicians  with  that  of 
Inexperienced  Technicians,  for  Parts-Ordering  Interval 
(Average  Times  in  Minutes  Required;  Relay  Tests  and  Multiple  Tests) 


RELAY  TEST 


16.00 

16.00 

14.00 

12.00 

10.00 

6.00 

6.00 

4.00 

2.00 

0.00 


le.bo 


9.25 


— . i 


11.69 

j _ 

lETM 


Paper 


[.  I  EXPERIENCED 
□  INEXPERIENCED 


MULTIPLE 


[  J  EXPERIENCED 
□  INEXPERIENCED 
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MUTES 


F  I  0  U  R  B  8 

Compariaon  of  Performance  of  Techniciana  Using  lETMs  with  that  of 
Technicians  Using  Paper  TMs,  for  Parts-Ordering  Interval 
(Average  Times  in  Minutes  Required;  Relay  Tests  and  Multiple  Tests) 


TABLE  9 

Effect  of  Using  lETMs  Instead  of  Paper  TMs 
in  Parts-Ordering  Interval 


AVERAGES  OF  INDIVIDUAL  RATIOS  OF 

PERFORMANCE  TIME  (lETM) 

’  PERFORMANCE  TIME  (PAPER) 

for  each  technician 


RELAY  TESTS 

Experienced  Technicians 

Inexperienced  Technicians 

0.824 

0.761 

MULTIPLE  TESTS  | 

Experienced  Technicians 
inexperienced  Technicians 

______ 
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Hie  Maintenance  Close-Out  Interval  started  when  the  Technician  had  convicted  a  System-Health 
check  (or  with  the  declaration  of  a  CND  status  for  CND  faults)  and  ended  when  the  Close-Out  portion 
of  tlie  VIDS/MAF  was  conplete  (see  section  3.8.4).  Table  10  presents  the  mean  performance  times  for 
this  Interval  b}’  fault  type,  by  experience  level  and  by  medium  Figures  9  and  10  show  performance  times 
for  this  Interval  with  comparisons  based,  respectively,  on  the  two  oqierience  levels  of  the  Technicians  and 
on  the  two  types  of  medium.  Table  1 1  shows  the  ratios  of  perfonnance  times  for  the  Maintenance  Close- 
Out  Interval  with  lETM  use  to  those  with  paper-TM  use. 


4.1. l.S  Oveiall  Performance  Times 

The  Fault-Isolations  that  the  Technicians  were  asked  to  perform  included  seven  task  Intervals;  the 
Overall  (total)  Performance  Time  is  the  sum  (for  each  Technician)  of  the  times  for  four  of  these  Intervals 
(Preparation,  Fault-Isolation,  Parts-Ordering,  and  Maintenance  Close-Out).  See  sections  3.8.4. 1  and 
3.8.4.2.  Tables  12-17  present  the  Overall  performance  times  for  ea^'h  of  the  four  Intervals  analyzed  and 
the  Overall  times  for  the  total  test  sequence;  by  fault  type,  by  experience  level,  and  by  medium. 
Figures  11  and  12  compare  Overall  performance  times,  with  conq)arisons  based,  respectively,  on  the  two 
experience  levels  of  the  Technicians  and  on  the  two  typra  of  medium  Table  1 8  summarizes  Overall  test 
performance  times  with  lETM  siqjport  and  with  paper-TM  svqjport  for  each  lest  type  and  provides  the 
performance-time  ratios  for  each  Technician.  Table  19  sunirnarizes  the  ratios  ofOverall  performance  time 
with  ffiTM  support  to  that  with  paper-TM  siqjport. 


4.1.2  Perfonnance  Errors 

Performance  ©tors  made  by  the  Technicians  are  of  three  types: 

a.  Procedural  Errors. 

b.  False  Removals  (removal  of  good  conqxinents  thou^t  to  be  faulty). 

c.  Failures  to  Identify  the  Fault,  or  to  identify  the  problem  as  a  CND  (Failure  to  Fault  Isolate) 
Numbers  of  each  type  of  eiror  me  presented  in  the  following  sections,  and  sunimaraed  in  Table  20. 
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TABLE  10 
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PIOURB  9 

Comparison  of  Performance  of  Experienced  Technicians  with  that  of 
Inexperienced  Technicians,  for  Maintenance  Close-Out  Interval 
_ (Average  Times  in  Minutes;  All  Tests) _ 

RELAY  TEST 


4.B3 

4.13  I 


3,13 


1.B4 


IBTM  Papar 


□  EXPERIENCED 

□  INEXPERIENCED 


C  ND  TESTS 


4.0B 


4.10 


□  EXPERIENCED 

□  INEXPERIENCED 


MULTIPLE  TESTS 
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FZaURB  10 


Comparison  of  Performance  of  Technicians  Using  lETMs  with  that  of 
Technicians  Using  Paper  TMs,  for  Maintenance  Close-Out  Interval 
(Average  Times  in  Minutes;  All  Tests) 


TABLE  11 

Effect  of  Using  lETMs  Instead  of  Paper  TMs 
In  Preparation  Interval  (All  Tests) 

AVERAGES  OF  INDIVIDUAL  RATIOS  OF 

PERFORMANCE  TIME  (lETM) 

’  PERFORMANCE  TIME  (PAPER) 

for  each  technician 


RELAY  TESTS 

Experienced  Technicians 

Inexperienced  Technicians 

0.563 

0.836 

Experienced  Technicians 

Inexperienced  Technicians 

0.640 

1.011 

MULTIPLE  TESTS  | 

Experienced  Technicians 

Inexperienced  Technicians 
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AVERAGE  TOTAL,  INEXPBWNCED  TECHNICIAIiS: 


TABLE  14 

TOTAL  TIMES  REQUIRED  FOR  CND  TESTS  USING  lETMi 
(ALL  TECHNICIANS) 


I  TECHNICIANS  |  INTERVALS  |  TOTALS  "j 


PREPARATION 

FAULT- 

ISOLATIQN 

MAINTENANCE 

CLOSE-OUT 

S1 

14 

55 

2.5 

71.60 

82 

2 

36 

5 

43.00 

S3 

3 

51.5 

5 

59.60 

84 

3 

48 

4 

55.00 

85 

1 

26 

1 

28.00 

86 

3 

26 

1 

30.00 

87 

2 

22 

3 

27.00 

812 

3 

43 

1 

47.00 

AVERAGE  TOTAL,  EXPERIENCED  TECHNICIANS: 

45.13 

I 

1 

PREPARATION 

FAULT- 

ISOLATION 

MAINTENANCE 

CLOSE-OUT 

1 

i 

89 

3 

55 

7 

810 

3 

66 

3 

811 

2 

52 

2 

66.00 

813 

1 

40 

5 

46.00 

814 

2 

26 

1 

29.00 

816 

1 

46 

1 

48.00 

817 

3 

39 

3 

45.00 

818 

2 

25 

5 

32.00 

AVERAGE  TOTAL,  INEXPERIENCED  TECHNICIANS: 

47.88 

Note;  The  CND  Tests  did  not  Include  a  Parts-Ordering  Interval. 
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TABLE  18 

TOTAL  TIMES  REQUIRED  FOR  CND  TESTS  USING  PAPER  TMs 

(10  TECHNICIANS) 


L  h  - 

■UiQjlSXi 

FAULT- 

MAINTENANCE 

PREPARATION 

ISOLATION 

CL08E-0UT 

82 

6 

38 

5 

48.00 

84 

4 

60 

3 

67.00 

85 

5 

52 

5 

62.00 

86 

3 

68 

3 

74.00 

87 

S 

41 

4 

50.00 

AVERAGE  T( 

OTAL,  EXPERIENCED  TECHNICIANS: 

60.20 

FAULT- 

MAINTENANCE 

811 

A 

31 

2 

2 

46 

2 

7 

41.5 

4.5 

817 

5 

62 

6 

818 

4 

42 

6 

52.00 

AVERAGE  TOTAL,  INEXPERIENCED  TECHNICIANS: 

53.00 

Note;  The  CND  Tests  did  not  include  a  Parte-Ordering  Interval. 
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TABLE  16 

TOTAL  TIMES  REQUIRED  FOR  MULTIPLE  TESTS  USING  lETM* 
(ALL  TECHNICIANS) 


i  TECHNICIANS  |  -INTERVALS-  |  TOTALS  | 


PREPARATION 

FAULT- 

JSOLATJQN 

PARTS- 

ORDERING 

VIAINTENANCE 

CLOSE-OUT 

S1 

2 

61 

12 

1 

66.00 

S2 

2 

56 

8 

2 

68.00 

S3 

2 

102 

7 

1 

112.00 

S4 

3 

46 

12 

1 

61.00 

S5 

1 

26 

8 

1 

36.00 

S6 

2 

64 

9 

1 

76.00 

S7 

5 

71 

10 

2 

88.00 

812 

2 

44 

8 

3 

67.00 

AVERAGE  TOTAL,  EXPERIENCED  TECHNICIANS; 

70.80 

ESEPARAIIQN 

FAULT- 

ISOLATION 

PARTS- 

ORDERINQ 

MAINTENANCE 

CLOSE-OUT 

S9 

4 

56 

16 

4 

79.00 

810 

2 

71 

14 

1 

88.00 

S11 

2 

66 

13 

3 

83.00 

S13 

4 

66 

10 

3 

73.00 

814 

2 

59 

18 

0.6 

70.60 

S16 

1 

47 

10 

4 

62.00 

S17 

2 

39 

10 

1 

52.00 

S18 

1 

90 

16 

2 

109.00 

AVERAGE  TOTAL,  INEXPERIENCED  TECHNICIANS; 

78.19 
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FISUKK  11 

Comparison  of  Performance  of  Experienced  Technicians  with  that  of 
Inexperienced  Technicians,  for  Total  Tesc  Performance  Times 
(Average  Total  Times  in  Minutes  Required;  All  Tests) 


RELAY  TEST 


80  0 
70.0 
80.0 
SO.O 
40.0 
30.0 
20.0 
10.0 
0.0 


78.8 


60.8 

B9.e 

B1.4 

[J  EXPERIENCliiD 
□  INEXPERIENCED 


lETM 


P  u  p  •  r 


FIGURE  12 

Comparison  of  Performance  of  Using  lETMs  with  that  of 
Technicians  Using  Paper  TMs,  for  Total  Test  Performance  Times 
(Average  Total  Times  in  Minutes  Required;  All  Tests) 
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TABLE  19 


Effect  of  Using  lETMs  Instead  of  Paper  TMs 
on  Total  Test-Performance  Times 

SUMMARY  OF  RATIOS  OF 
INDIVIDf 7AL  TECHNICIAN  PERFORMANCE  TIMES 
FROM  TABLE  18  (ALL  TESTS) 


RELAY  TESTS 

Experienced  Technicians 

0.863 

Inexperienced  Technicians 

0.862 

Experienced  Technicians 

Inexperienced  Technicians 

0.623 

0.734 

WESfimamm 

Experienced  Technicians 

Inexperienced  Technicians 

0.690 

0.636 
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INCIDENCE  OF  PROCEDURAL  ERRORS 
CALL  TESTS) 
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4, 1.2.1 


A  Procedural  Error  is  defined  as  a  serious  misuse  of  the  procedures,  e.g.,  technician  may  perform  a 
test  on  the  wrong  pin  and,  as  a  result,  branch  to  an  imiKtper  sequence.  Table  20  presents  the  number  of 
these  types  of  errors  as  a  flmction  of  technician  experience  level,  of  fault  type,  and  of  medium 


4. 1.2.2  Success  in  Isolating  the  Fault 

Ihe  lEIMPMA  medium  was  used  to  si^iport  Fault  Isolation  in  48  cases;  the  paper  medium  was  used 
in  47  cases.  (One  of  the  tests  involving  a  paper  TM  [Technician  SIO,  CND  test]  was  voided  because  of 
a  test  protect.'!  cnor.)  In  tlie  Relay  and  Multiple  Tests,  all  Fault  Isolations  were  successfiilly  performed 
by  both  Expenenced  and  Inexperienced  Technicians  with  both  TI  media.  For  the  Q4D  tests,  the  correct 
result  was  a  determination  tliat  the  syn^oms  presented  could  not  be  diqslicated;  using  the  lEIMPMA 
combination,  all  Technicians  arrived  at  the  correct  result;  with  paper  TMs,  three  of  the  Ejqserienccd 
Technicians  and  two  of  the  Inejqserienced  Technicians  failed  to  arrive  at  the  correct  result.  Thus,  there 
were  a  total  of  S  failures  to  Fault-Isolate  out  of  95  tests,  oil  using  the  paper  TM. 

4. 1.2.3  False  Removals 

In  Fault  Isolations  and  Corrective  Maintenance,  a  False  RenfX)val  is  declared  \\hen  a  technician 
replaces  and  then  sends  to  the  Intermediate  Maintenance  Activity  (EMA)  a  part  he  believes  to  be  faulty 
but  in  fact  testing  at  IMA  shows  the  part  to  be  good  EXiring  this  Test,  \\hen  a  Technician  identified  a 
part  that  he  believed  to  be  the  cause  of  the  fault  but  in  fact  was  not,  the  technical  observer  "simulated" 
the  replacement  and  told  the  Technician  to  proceed  with  usual  practice,  e.g.,  a  "System-Health  Check". 
Of  course,  the  check  showed  the  same  symptom;  where^xin  the  technical  observer  declared  a  False 
Removal  and  the  observer  directed  the  Technician  to  resume  troubleshooting.  lEIMPMA  users  and  the 
paper  TM  users  committed  3  False  Removals  each.  The  incidence  of  these  False  Removals  is  shown  on 
Table  20. 
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5.0  EVALUATION  OF  PERFORMANCE  RESULTS 


5.1  GENERAL  CXlNSmERAHONS 

As  stated  in  section  1,  this  field  test  yaa  carried  out  to  evaluate  a  number  of  innovations  in  lEIM 
technology  \\hich  have  been  developed  and  evaluated  in  the  laboratoiy  under  the  USAF  IMS  program 
(see  section  1.3).  The  final  assessment  of  any  such  technological  advances  depends,  of  course,  on  their 
use  by  active  technicians  at  an  operational  base  using  operational  aircraft.  Measures  of  their  effectiveness 
in  the  field,  as  noted  in  section  4.1,  are; 

a.  Conparisons  of  (1)  times  required  to  carry  out  specific  Fault-Isolation  tasks  sipported  by  an 
lETM  with  (2)  performance  times  vviien  the  same  technician  (or  one  with  similar  eoperience) 
carries  out  the  same  type  of  Fault-Isolation  task  using  a  paper  TM  for  sipport  of  his  effort. 

b.  Conparisonof(l)  number  ofetrorsoccuning  when  the  technicians  are  supported  by  lETMs  with 
(2)  the  numba  of  errors  occurring  when  the  technicians  are  sipported  by  paper  TMs. 

Additionally,  in  evaluating  the  usability  and  operational  suitability  of  such  innovations,  a  careful 
recording  of  technicians'  reactions  is  critical.  Thus,  even  thou^  a  given  ipproach  may  be  made  to  work 
in  the  field  (in  a  specific  test),  it  must  be  abandoned  or  modified  if  it  makes  the  technicians'  job 
significantly  more  difficult,  or  significantly  Increases  training  requirements.  Moreover,  such  comments 
are  of  great  inportance  in  pointing  out  approaches  to  inproving  the  processes  and  design  of  the  processes 
under  evaluatioa 


5.1.1  Siuumary  of  Previous  Teat  Results 

As  noted,  previous  field  tods  performed  the  Navy  and  Air  Force,  with  a  variety  of  hardware 
systems,  have  consistently  shown  significant  inprovement  in  technician  performance  when  TT  was 
displayed  in  lETM  form  In  every  case,  technicians  were  overwhelmingly  in  favor  of  the  lETM  pproach 
as  conpered  with  the  use  of  pper  TMs,  even  thou^  in  every  case,  a  number  of  criticisms  and  proposals 
for  inprovement  of  the  XERvI  were  provided. 
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Qualitatively,  the  following  principles  were  denwnstrated  by  these  tests: 

a.  Inpovement  in  performance  was  greater  for  inejqjerienced  technicians  than  for  ejqperienced 
technicians. 

b.  Improvement  in  performance  was  greater  for  more  complex  systems  and  procedures  (e.g.,  Fault 
Isolation)  than  for  sinpler,  more  stnd^tforward  procedures  (e.g.,  a  simple  remove>and-replace 
Corrective-Maintenance  action). 

Consequently,  the  data  have  been  displayed  in  section  4  to  conpare  the  effects  of  adopting  lEIMs 
as  a  function  of  technician  e7q)erience. 

5.1.2  Limitations  on  Interpretatioiu  of  Results 

In  essentially  all  populations  of  test  subjects,  there  were  one  or  two  subjects  whose  performance  was 
much  worse  than  the  rest,  and  one  or  two  whose  performance  was  much  better  than  the  rest.  Such 
atypical  conpetence  is,  of  course,  common  in  real  situatians.  In  the  present  tests,  to  make  behavior  of 
this  type  more  visible,  each  individual  result  of  each  of  the  16  Technicians  has  been  displayed  for  each 
ofthe  six  tasks  performed  Average  performance  times  for  each  groip  also  have  been  displayed,  but  the 
wide  distribudon  noted  in  this  limited  population,  althou^  conclusive  in  denronstrating  aygiali 
inprovement  in  performance,  calls  into  question  the  validity  of  using  these  data  in  a  statistical  or 
quantitative  sense  for  predictive  purposes. 

Similarly,  of  course,  the  differences  in  the  types  of  tests  performed,  in  the  types  of  faults  introduced, 
in  the  quality  of  the  lETM  material  prepared  for  each  test,  and  in  technician  erqwrience,  makes  inpossible 
the  use  for  general  predictive  purposes  of  data  averaged  over  more  than  a  single  test  grotp  (with  its  unique 
combination  of  technician  experience,  test  type,  test  interval,  and  medium). 

5.13  Summary  of  Technician  Perfomumoe 

The  most  significant  test  Intervals  were  the  Fault-Isolation  Interval  and  the  Paits-Qrdering  Interval. 
Tables  16  aixi  17  show  the  factions  of  total  test  time  occupied  by  these  test-interval  tinnes,  for  all  test 
subjiXta  and  for  all  tests. 
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S.1.3.1  Relative  Fault-Isolation  Effectiveness 


For  the  Fault-Isolation  Interval,  Table  6  sho\vs  relative  performance  times  of  Technicians  using 
lETMs  as  conpued  to  those  using  paper  TMs,  for  each  test  groiq). 

In  every  case,  performance  times  were  reduced  by  the  introduction  of  lETMs,  except  for  the  groi^ 
of  Eq^erienced  Technicians  performingthe  Relay  Tests,  in  wiiich  performance  times  were  essentially  equal 
for  the  two  media.  The  relative  performance  times  for  the  two  media  for  all  test  grotps  are  displayed  in 
Figure  5.  Note  that  for  Ine7q)erienced  Technicians,  introduction  of  lETMs  reduced  Multiple-Test  Fault- 
Isolation  time  by  about  one  third. 

Figure  5  shows  that  in  four  of  the  six  test-type/medium  combinations,  Experienced  Technicians 
perfoimed  the  troubleshooting  assignment  in  less  time  that  the  Inexperienced  Technicians;  but  in 
performing  the  CND  tests  with  paper  TMs,  Ejqxrienced  Technicians  took  about  16%  longer  than  did 
Inejqjerienced  Technicians. 


5. 1.3.2  Relative  Parts-Qrdering  Effectiveness 

It  is  noted  that  for  the  Parts-Ordeiing  Test  Interval,  the  performance  times  consisted  of  the  times 
required  to  locate  the  infonnation  needed  to  order  the  pelts  and  to  conplete  the  (qjprcpiate  section  of  the 
VTDS/MAF  form.  However,  in  the  case  of  the  lEIM/FMA  combination,  the  time  involved  in  the 
automated  completion  of  the  fonn  was  sufficiently  brief  and  constant,  as  compered  with  the  time  required 
to  assemble  the  information,  that  it  was  not  reported  In  the  case  of  the  pper-TM,  the  form-preparation 
portion  of  the  Parts-Ordering  Interval  (which  was  included  in  the  recorded  Parts-Ordering  performance 
times)  was  a  significant  part  of  the  overall  time  required  Any  direct  conperisons  of  performance  time 
should,  therefore,  be  made  with  this  factor  in  mind 

For  the  Parts-Ordering  Interval,  Table  8  shows  that  average  performance  times  were  decreased  in 
every  case  in  going  fixim  paper  TMs  to  lETMs. 

In  the  case  of  the  Miltiple  Tests,  Table  8  shows  that  average  performance  time  for  Bqierienced 
Technicians  was  reduced  by  more  than  50%. 
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5.2  TECHNICIANS'  RATING  OF  EFFECTIVENESS  OF  lETlVFPMA  FEATURES 

As  described  in  section  3.8.  le,  all  sbdeen  Technicians  (eight  Inexperienced  and  eight  bjqserienced), 
after  test  performance,  filled  out  User  Evaluation  (Questionnaires  in  which  they  rated  the  effectiveness  of 
various  features  ofthelETM/PMA  combination.  The  forms  used  are  reproduced  in  Appendix  D.  Features 
rated  were  of  two  types: 

a  Physical  Features  of  the  IHIM^MA  Combination  (IS  Items).  [See  section  5.2.1] 
b.  Sofhvare/pperational  Features  (25  Items).  [See  section  5.2.2] 

Each  of  these  features  was  rated  by  each  Technician  according  to  the  scale  shown  in  Table  21. 


TABLE  21.  Rating  Scale  Used  in  Technicians'  Evaluation  of 

lETMPMA  Physical  Features  and  Software/Operational  Features 

Seale  Value 

Scale  Definition 

0 

Unsatisbctory 

1 

Maiginal 

2 

Sadsfiictory 

3 

Highly  Sadsfiictory 

4 

Outstanding 

Can't  Evaluate 

Technicians'  ratings  were  averaged  separately  for  the  Inesqxdenced  Group  and  for  the  Experienced 
Qroif).  Thus,  an  average  rating  of  0  would  in^iy  that  all  the  members  of  the  group  found  the  particular 
feature  Unsatisfactory,  the  lowest  possible  rating,  whereas  a  rating  of  4,0  would  indicate  that  all 
Technicians  found  the  feature  Outstanding.  Tedmician  responses  were,  of  course,  subjective,  but  were 
based  on  Technicians'  opinions  as  to  the  suitability  of  a  given  feature  in  contributing  to  the  Fault  Isolation 
and  Nblntonance  reporting  tasks  they  had  just  performed. 


5.2.1  Physical  Features  of  the  lETM/PMA  Combination 


Table  22  presents  ratings  (scale  value  averages)  for  the  IS  Physical  Features  of  the  lEIMPMA 
\\hich  >vere  evaluated 

Table  23  summarizes  the  distribution  of  rating  for  Inexperienced  and  Ejqserienced  Technicians  for 
the  IS  features  evaluated 

Note  that  all  features  were  rated  at  kaat  Satisfactory  by  all  Technicians,  with  67°/o  (by  lne7q«rienced) 
and  87%  (by  Etqperienced)  rated  as  Hl^y  Satisfactory  or  between  Hi^y  Satisfactory  and  Outstanding. 

The  lowest  rated  features  (between  Satisfactory  and  Highly  Satisfactory)  were: 


4.  £ase  of  positioning/repositioning  PMA  at  worksite 

2.75/3.0. 

9.  Response  time  after  key  press 

2.125/2.375 

10.  Appropriateness  of  function  keys 

2.87/3.0 

14.  Olare  on  display  screen 

2.625/2.71 

15.  Key  pressure  resistance/sensitivity 

2.625/3.125 

Dissatisfhction  with  these  features  (amplitied  by  Technicians'  oral  and  written  comments)  clearly 
indicates  the  need  for  the  incorporation  of  functional  inpnvements  in  certain  features  of  the  lEIMPMA 
design. 

Response  time  (Feature  9)  was  on  occasion  as  high  5*8  seconds.  Dissatisfaction  with  such  a  delay 
time  in  displaying  of  the  next  screen  has  bcm  expressed  on  all  previous  field  tests,  but  the  incorporation 
of  other  functional  inpovements  in  the  version  of  the  hardware  and  software  used  during  this  test 
unfortunately  failed  to  correct  this  condition  Response-time  specifications  for  an  lETM/PMA 
combination  require  response  (flill  display  of  next  screen)  in  no  greater  than  1  second 

The  Olare  rating  referred  to  use  of  the  Liquid  Crystal  Display  with  the  backlighting  turned  off.  A 
position  switch  which  provided  screen  backlighting  eliminated  the  problem  Data  were  taken  with  screen 
lighting  on  or  off,  as  considered  desirable  by  the  individual  technicians. 
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5.2.2  SoftwarC'Operatioiuil  Features  of  the  IETM/PM4  Combination 

Table  24  presents  the  averaged  rating  for  the  25  lEIMPMA  Softwaie/Operational  Features 
evaluated 

Table  25  summarizes  the  distribution  of  ratings  for  Inexperienced  and  Exqxerienced  Technicians  for 
the  25  features  evaluated. 

M  features  yxxe  rated  kaat  SAtiafiietnry.  vvith  72%  of  them  rated  as  Hi^y  Satisfactory  or 
between  H^y  Satisfactory  and  Outstanding,  b>’  both  Irtexqserienced  and  Exqxerienced  Technicians. 


The  features  rated  lowest  (Satisfactory  or  between  Satisfactory  and  Hi^y  Satisfactory) 

by  both  grotpi  of  Technicians  were: 


22.  Ease  of  moving  cursor  with  arrow  keys 

2.87/3.25 

24.  Ease  of  moving  cursor  with  thumb  knob 

2.00/2.50 

25.  Ease  of  returning  to  appropriate  place  in  a  set  of 

procedures aAer branching else\^here in  database 

2.00/2.50 

27.  i^xpropriate  number  of  procedural  steps  per  screen 

2.87/2.625 

29.  Scrolling  function  availability 

2.875/2.86 

30.  Scrolling  mode  (hard  key) 

2.875/2.83 

31.  Scrolling  with  arrow  keys  and  SELECT  key 

3.00/2.86 

35.  Cursor  visibility 

2.75/3.125 

38.  Ease  of  accessing  locator  diagrams 

3.25/2.875 

40.  Adequacy  of  wiring  diagrams 

3.105/2.875 
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Aithcu^  as  noted,  all  of  these  features  were  at  least  Satisfactory,  Technicians  were  less 
enthusiastic  about  the  following  aspects  of  the  lElMPMA  combination  than  they  were  about  other 
lEIM/PMA  software/opcrational  factors: 

a.  Cursor  operation  (22, 24, 35).  Careful  attention  will  be  given  in  future  PMA  designs  to  cursor 
configuration  and  operation  to  inpove  this  aspect  of  the  PMA  operatioa 

b.  The  Scrolling  fhnctiun  (29, 30, 31).  The  introduction  of  a  scrolling  function  into  a  small  PMA 
is  a  difificult  design  problem,  both  fiom  the  strnc^int  of  the  mechanics  to  produce  the  scrolling 
and  fiom  the  standpoint  of  satisfactory  display  (e.g.,  of  drawings  or  diagrams  too  large  for  single¬ 
screen  display).  Work  is  being  carried  out  both  on  inprovement  of  scrolling  itself,  and  on  the 
development  of  display  methods  which  will  obviate  the  need  for  scrolling  altogether. 

c.  Special  aspects  of  lETM  design  (27, 38,  39).  These  ratings,  together  with  a  number  of  oral 
comments  concerning  various  aspects  of  the  quality  of  the  displayed  Technical  Information, 
indicate  the  need  for  inprovement  of  TI  quality  as  it  relates  to  small-screen  display,  especially 
in  the  areas  of  graphics  and  text-gnphics  interfaces.  At  this  time,  there  appears  to  be  a  trade-off 
between  the  extent  of  the  use  of  automation  in  preparing  the  Tl  (a  number  of  approaches  to  Tl 
automation  were  field-tested  for  the  first  time  during  this  test)  and  the  user-ffiendliness  of  the 
finished  lETM.  The  lEIM-picparation  process  must  be  refined  to  optimize  BETM  utility,  even 
at  the  cost  of  more  conplex  automated  preparation  techniques  or  less  automated  procedures 
requiring  greater  author  involvement  in  the  process. 

d  Ability  to  return  to  a  specific  point  in  the  procedure  after  branching  (2^.  (In  spite  of  this 
relatively  low  rating,  no  comments  concerning  this  feature  were  made  during  the  oral  debriefing 
sessions.)  This  c^sability  will  be  reviewed  to  determine  whether  problems  exist  in  this  function 
as  designed;  i.e.,  whether  the  PMA  fails  to  return,  or  returns  slowly,  to  its  prebranch  point  on 
request. 


53  EVALUATION  OF  KEY  ELEMENTS  OF  THE  UTMFMA  COMBINATION 

This  Section  summarizes  technical  evaluations  repcPed  by  both  Test  Observers  and  Technicians  on 
various  aspects  of  the  lEIM/PMA  design  and  operatiorL 
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5.3.1  Tlie  BIT>]ETM/PMA  Interlace  Using  the  1553  Bus 


The  BIT-IEIM^MA  interface  allows  the  technician  to  use  the  PMA  to  interact  with  and  control  the 
F/A-18  BIT  for  the  Eight  Control  System  This  BIT  control  capability  requires  the  technician  to  install 
an  electrical  cable  connecting  the  PMA  to  tttt  aircraft's  1 553  multiplex  bus.  During  the  test,  the  main  uses 
of  this  capability  involved  the  Memory  Inspect  (MI)  and  Test  Qroqp  (TG)  procedures. 

Two  bases  were  ased  to  evaluate  this  direct  BIT  access  capability  of  the  PMA; 

a.  Test-observers'  evaluations  of  its  effectiveness  during  the  48  Fault-Isolation  tasks  in  which  it  was 
used 

b.  Technicians'  debriefing  comments  on  the  lEIMPMA-BIT  interface. 

5.3.1.1  Summary  of  Observers'  Evaluations 

The  lETMTMA  control  of  BfT  worked  well  throughout  the  test,  including  the  hoolaq)  and  operability 
test  as  well  as  its  use  in  conducting  TG  and  MI  procedures.  The  following  occurrence  is  an  example  of 
its  utility;  One  of  the  Fault-Isolation  procedures  called  for  a  particular  TG  procedure  whose  outcome 
should  have  been  a  particu%  BIT  Logic  Inspection  (DLIN)  code.  However,  after  BIT  ran  the  TG 
procedure  commanded  by  the  lETMPMA,  the  PMA  displayed  a  BUN  code  different  fiom  the  expected 
one.  Suspecting  a  problem  with  the  lEIMPMA  or  the  1553  interface,  the  interface  was  disconnected  and 
the  TG  procedure  was  rerun  using  the  aircraft's  DDIs  and  FCCs  (neither  the  PMA  nor  its  interface  with 
BIT  was  active).  The  test  result  without  the  PMA/1553  interface  was  the  same  as  with  IETM/PMA/1553 
interface  meaning  that  the  aircraft  had  a  iwn-test  fault  (an  activator  failure)  and  the  lETM/PMA  working 
with  the  FCS's  Bfl  throu^  the  1553  interface,  had  detected  it  Instances  of  a  similar  nature  occurred, 
involving  generator  oveibeat  and  pin  damage  in  cable  connectors.  These  instances  are  interpreted  as 
evidence  that  the  Bn'-1553-IETMPMA  interface  provided  valid  results. 
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5.3.1.2 


Technicians'  overall  opinion  of  the  1 553  interface  was  positive  althou^  some  suggestions  were  made 
for  inpovcment.  (Technicians'  comments  on  all  phases  of  the  Test  are  presented  in  detail  in 
i^jpendix  A.) 

a.  The  positive  aspects  included: 

(1)  Data  displays  were  more  readable,  and  less  cluttered  on  the  PMA  than  on  the  aircraft's  DDIs. 

(2)  Automated  assessments  of  Memory  Inspect  numbers  were  far  easier  and  far  more  accurate 
than  the  manual  assessments. 

b.  Negative  aspects  included: 

(1)  The  electrical  cord  was  an  inqxdiment  to  Technician  mobility. 

(2)  The  process  via  the  PMA/BIT  interface  was  slower  than  the  direct  manipulation  of  cockpit 
controls. 

(3)  Mormation  to  be  cornpaiedwesdispkyed  on  two  screens  introducing  unnecessary  difficulty. 

SJ.2  The  Portable  Maintenance  Aid  (PMA) 

A  I-himan  Factors  review  of  the  PMA  hardware  raised  24  issues  regarding  the  PMA  design.  The 
more  troublesome  issues  are  noted  below: 

a.  It  is  difficult  to  move  the  cursor  among  non-acyacent,  nonaligned  areas. 

b.  There  are  too  many  methods  for  moving  the  cursor  and  pointer. 

c.  The  utility  of  user-controllable  font  size  is  questionable, 
d  Viewing  angle  is  restricted  by  the  PMA  bezel. 

e.  Confusion  exists  among  the  alternative  methods  of  selecting  opticffis  displayed  on  the  screen. 
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5.3.3  lETM/PMA  Presentation  Features  Considered  Effective 


a.  Integration  of  Information; 

(1)  Side-by-side  text  and  si^jporting  graphics. 

(2)  Linking  to  cross  references  to  branching  procedures. 

(3)  Ability  to  call  locator  graphics;  e.g.,  accessed  via  Required  Conditions  or  a  soft  key. 

b.  Automated  Functions: 

(1)  Automation  of  VIDS/MAF  completion  (especially  the  Parts-Ordering  section). 

(2)  Ability  to  perform  Memory  Inspect. 

(3)  Control  of  the  FCS  BIT  via  the  1553  multiplex  bus. 

c.  Error  Prevention/Reduction; 

(1)  Regaining  place  in  a  procedure  after  interri^oa 

(2)  Dropping  a  row  vthen  moving  to  the  rijS^it  to  get  the  "Go  To"  instruction. 

d.  PMA  Portability  (Size  and  Wei^). 

e.  Level  of  H  detail. 

t  Displays  only  that  information  relevant  to  the  Technician's  assigned  aircraft. 

53.4  Suggestioiu  to  Further  Improve  lElMFMA  (not  in  order  of  inportance) 

a.  Provide  a  browse  mode  for  work  planning  (i.e.,  a  capability  for  a  Technician  to  scan  through 
maintenance  procedures  available  in  the  EETM/PMA,  and  review  informatioivaccess  paths  and 
navigation  instructions). 

b.  Speed  PMA's  response  time. 

c.  Provide  procedures  for  pre<xpended  parts. 
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d.  Provide  additional  locator  information  on  wiring  diagrams. 

e.  Provide  both  abbreviated  wiring  diagrams  and  schematics. 

C  Provide  lockout  to  prevent  inadvertent  key  activation. 

g.  Reduce  screen  glare  when  not  "backlit”. 

h.  Make  PMA.  less  awkward  in  cockpit. 

L  1553  interface; 

(1)  Should  be  t^lied  so  as  not  to  disable  the  mission  conputers  which  in  turn  disabled  the 
Digital  Display  Indicators  (DDIs). 

(2)  Improve  the  Nosewheel  Well  location  of  the  interface  plug. 

(3)  Design  so  that  the  1553  umbilical  cord  does  not  affect  PMA  portability. 

(4)  Consolidate  feed  from  the  data  bus  to  the  PMA  into  one  vs  two  screens. 

J.  Standardize  location  of  watch  icon 

L  Oin^iate  cursor  expectancy  violations. 

L  Provide  alerts  for  iqxoming  series  of  checks, 
m.  Ease  access  to  inportant  informatioa 

(1)  Initial  set  \xp. 

(2)  Fault  identificatiQn. 

(3)  Completing  the  VIDSM^F. 

D.  Standardize  on  one  means  of  cursor  control. 

0.  Use  a  1<0  (not  the  0-9)  sequence  for  number  keys. 

p.  EstablishccnisistencywithopeFBticnialpractices;e.g,thoseinvolvingexlernalhydraulicgenerators 
and  preperadon  of  the  VIDS/MAFs  for  the  multiple  faults. 
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q.  Design  friendly  abort  and  restart  procedures. 

r.  Arrange  so  that  there  is  no  lEIMPMA  advance  until  CDI  sign  off. 

s.  Eliminate  confusing  similarities  among  required  conditions,  follow-on  and  closing  actions, 
t  Provide  more  precise  uses  for  the  diagnostic  block  diagram. 

u.  Ruggedize  the  PMA;  increase  the  PMA  capability  of  withstanding  harsh  environmental  conditions. 
V.  Provide  better  labelling  of  soft  function  keys, 
w.  Provide  better  battery  life  and  recharge  procedure. 

X.  During  a  maintenance  procedure,  present  a  record  of  the  time  spent  on  the  procedure  so  far,  as 
compared  with  norm  or  total. 

y.  Sinplify  the  highlight  and  select  functions. 

Zi  Reduce  Redundancy  in  Function  Keys. 

aa.  Suggestions  for  Additional  Integration.  Although  the  Savy  and  Air  Force  are  already  pursuing 
many  of  the  Technicians'  suggestions,  the  suggestions  are  reported  here  as  a  field  endorsement  of 
these  programs. 

(1)  Provide  link  to  Maintenance  Control  (MQ. 

(2)  Provide  next  da^s  flig^  schedule. 

(3)  Provide  an  interface  between  the  PMA  and  the  Data  Storage  Unit  (DSU). 

(4)  Add  training  content  to  the  lETM  Data  Base. 

5  J.5  SuitabiUty  of  the  PMA  for  Heet  Use 

RcviUlts  for  this  Test  have  shown  that  the  Air-Frace-developed  PMA  was  very  suitable  for  the 
application  for  which  it  was  intended.  However,  for  Fleet  use  (operation  of  aircraft  fix>m  carriers),  certain 
inpnvements  are  needed,  as  identified  by  the  users  cited  in  this  Section  and  in  i^Tpendbe  A;  e.g., 
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inpx)ved  cabling  for  external  connection  (1553  bus  or  external  power)  so  as  not  to  impede  device 
portability,  better  cursor/pointer  control  (such  as  a  mouse),  longer  battery  life  and  unobtrusive  battery 
charging,  less  glare  from  screen,  greater  ruggedness,  suggested  relabeling  of  function  keys.  Ihese 
conplaints  '.vill  generally  apply  to  any  portable  display  device  now  commonly  available  in  the  market 
place. 

Fleet  use  of  PMAs  would  in^se  additional  requirements  such  as  incorporation  of  an  interface  to  the 
Navy  SNAP  and  NALCOMIS  Programs,  a  diagnostic  application,  or  a  training  module,  all  of  which  have 
been  developed  by  other  activities.  Such  iqrplications  can  be  performed  with  available  commercial 
software  such  as  MS/DOS  and  MS/Wlndows,  operating  with  an  INTEL  X86  chip  set.  Ihe  PMA  used  in 
this  Test  enploys  a  Motorola  CPU  and  a  DOS-incompatible  operating  system.  Althou^  ruggedized  to 
some  extent,  the  Air  Force  PMA  does  not  meet  MIL-E-16400  requirements  for  the  hi^-humidity  and  salt- 
spray  environments  encountered  in  shipboard  operations. 

Thus,  Navy  requirements  for  PMAs  (PEDDs)  rai^  include  some  ruggedized  COTS  devices  (even 
less  rugged  than  the  Air  Force  PMA)  and  some  veiy  rugged  MELSPEC  devices,  effectively  bracketing  the 
ctqiability  of  the  AF  PMA  in  this  regard.  The  memory  capability  of  the  PMA,  both  RAM  and  nonvolatile, 
is  very  low  con^xired  with  available  commercial  nomts.  The  Navy  device  would  require  8  or  10  MByte 
RAM  and  several  hundred  MByte  nonvolatile  storage,  vs  the  6/32  MByte  allocation  of  the  Air  Force 
PMA.  The  packet  radio  on  the  Air  Force  device  is  undesirable  in  the  EMI-sensitive  environment  of  Navy 
use. 

In  summary,  the  factors  discussed  above  lead  to  the  conclusion  that  the  Air  Force  PMA,  although 
very  equable,  would  not,  in  its  present  form,  be  suitable  for  the  Fleet.  A  more  extensive  evaluation  would 
involve  additional  factors. 

5.4  TEOlMaAN  FREF1»ENCE  FOR  im  lETM/FMA  GOMBIIHATION  COMPAQ 
PAPER  TECHNICAL  MANUALS 

This  Section  summarizes  Technician  preferences  for  various  aspects  of  the  lEIMFMA  combination 
as  con^xued  with  the  NAVAIR-based  work-package  F/A-18  Technical  Manuals. 
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5.4.1  Evaluation  of  Technician  Preference  for  lEIMPMA  vs  Paper  TMs 


Section  3  of  the  User  Evaluation  Questionnaire  (see  Appendix  D)  consisted  of  a  rating  form  v^ch 
solicited  a  conparative  assessment  of  the  lETMPMA  ^vith  respect  to  the  standard  NAVAIR  pe^-based 
TMs.  Table  26  sho>vs  the  rating  scale  used. 


TABLE  26.  Rating  Scale  Used  in  Technicians'  Comparative  Evaluation 

of  lETM/PMA  and  Paper 

TM  Characteristics 

Scale  Value 

Scale  Definition 

0 

Paper  TM  Significantly  Better 

1 

Paper  TM  Slightly  Better 

2 

No  Difference 

3 

lETM/PMA  Slightly  Better 

4 

lETM/PMA  Significantly  Better 

MM 

Can't  Evaluate 

Technicians'  preference  was  evaluated  iiom  ei^  stan^ints  (questions  4148  of  the  User  Evaluation 
Questionnaire).  All  sbcteen  Technicians  (ei^  Ine?q)erienced  and  ei^t  Ejqjerienced)  filled  out  the  rating 
form.  Results  are  shown  in  Table  27  as  averaged  scale  factors  for  each  group. 

As  shown  in  previous  field  tests,  both  groups  of  Technicians  preferred  the  lEIMPMA  combination 
to  the  use  of  paper  IMs  for  use  in  F/A- 1 8  troubleshooting.  Particularly  ^ipealing  to  the  Technicians  is 
the  ability  to  obtain  required  Technical  Information  without  the  need  to  search  through  many  pages  (or 
volumes)  of  conventional  pe^-based  Technical  Manuals  (e.g.,  questions  41, 42, 43).  Technicians  were 
sonewhat  less  enthusiastic  about  the  actual  presentation  of  the  Tl  itself  (questions  44  [Ine?q)erienced],  47, 
48).  As  noted  in  section  S.2.2,  additional  work  is  required  to  optimize  for  the  user  the  actual  display  of 
Technical  Information,  and  e7q)loit  the  ctqxtbility  of  a  luminescent  screen  to  provide  a  Technician  with 
more  effective  maintenance  siqjport  informatics. 
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TABLE  27. 

Technicius*  Preference  for  lETM/PMA 


6.0  RECOMMENDATIONS 


As  a  result  of  the  present  evaluation  by  the  Navy  and  Air  Force  participants  in  the  preparation  and 
field  testing  of  IHTMs  based  on  the  latest  IMS  technology,  the  following  actions  are  recommended; 


6.1  REVIEW  OF  METHODS  FOR  AUTOMATED  PREPARATION  OF  lETMs 


6.1.1  Preparation  of  lETM  for  Test 

In  the  present  Test,  the  lEIM  provided  to  the  using  Technicians  was  prepared  throu^  creation  of 
an  lETM  Data  Base,  by  the  McDonnell  Aircraft  Co,  This  lETMDB  was  generally  in  accordance  with  the 
Air  Force  Content  Data  Model  (CDM),  as  described  in  the  DOD  lETMDB  Specification  MIL’D-87269 
(N\hich,  however,  was  not  yet  published  at  the  time).  The  actual  EBTM  material  (i.e.,  translation  of  the 
lETMDB  to  the  material  actually  displayed)  was  con^sed  throu^  the  use  of  techniques  that  were  almost 
entirely  automated,  with  the  software  (the  Presentation  System)  hosted  in  the  PMA  itself. 


6.1.2  Evaliuition  of  lETM  Preparation  Process 

Comments  as  to  the  effectiveness  of  the  lETM  preparation  process  were  obtained  fiom  throe  sources; 

a  The  oqwrience  of  the  AF  Contractor  in  preparing  the  DETMDB  used  (based  on  NAVAIR  F/  A-18 
paper  TMs), 

b,  The  experience  of  the  Armstrong  Laboratory  (AL/HRGO)  in  converting  this  material  to  iblM 
form,  and 

c.  Test  Technicians  and  Test  Observadcxi  personnel,  based  on  the  results  of  user  performance  with, 
and  preference  for,  the  Technical  Informatioa 
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6.U  RECOMMENDATIONS 


Based  on  evaluation  of  these  sources,  the  following  efforts  with  respect  to  lETM  preparation  are 
recommended 

a  lEIMDB  requirements  given  in  MIL-D-87269  should  be  carefully  reviewed  fix>m  the  standpoints 
of: 

(1)  adequacy  in  siqsporting  the  preparation  of  optimal  lEIMs,  either  by  means  of  automated 
techniques  or  by  an  author,  or  by  a  combination  of  both;  and 

(2)  practicality  for  preparation  of  the  IEIM)B  to  achieve  completeness  without  unnecessary 
effort  or  conplexity. 

b.  Available  procedures  for  preparation  of  lEIMDBs  in  accordance  with  MIL>D-87269  should  be 
carefully  reviewed  A  procedural  Guide  for  such  preparation,  for  the  guidance  of  Contractors  and 
DOD  System  Acquisition  Managers,  should  be  prepared  and  promulgated  (The  DOD-established 
Tri'Service  lEIM  Working  Groip  is  preparing  such  a  document,  conpledon  of  which  is  currently 
scheduled  for  the  first  quarter  of  FY  1995.) 

c.  F^vther  evaluation  is  required  to  determiriB  the  extort  to  which  lEIMDB  information  so  prepared 
can  be  extracted,  conpiled,  ordered,  and  fornif^ted  for  viewing  by  a  technician,  through  use  of 
a  Presentation  System  of  the  IMIS  type  (i.e.,  "dynamically",  by  the  software  hosted  in  the  PMA), 
without  loss  of  effectiveness  and  usa  fiiendliness.  (See  note  in  section  5.2.2.) 

d  Procedures  for  Validation  of  lETM  Technical  Information  in  the  lETMDB  and  H  at  the 
Presentation  System  level,  prepared  from  thr  lETMDB  by  automated  techniques,  should  be 
developed  in  accordance  with  MIL-Q‘87270  of  20  Nov  1992,  Qftaltiy  Assurance  Program: 
ItUeraOiveEUctronic  Techi^tiad  MantaUs  and  Associated  TechiUadlitformathn;  Requirements 
for  (ref.  16).  Such  procedures  should  be  standardized  and  incorporated  into  an  ipdated  version 
ofMIL-Q-87270. 
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6.2  IMPROVEMENTS  IN  PMA  CAPABIUTY 

Hie  effectiveness  of  a  PMA  in  aircraft  maintenarice  has  been  clearly  demonstrated  by  this  Test. 
Ifowever,  further  action  should  be  taken,  particularly  from  the  standpoint  of  human  factors  (user- 
interaction)  and  environmental/ruggedness  characteristics,  to  improve  the  device  to  make  it  more  effective 
in  operational  situations.  A  Tri-Service  set  of  standard  performance  requirements  for  the  PMA  (or  PEDD), 
similar  to  those  expressed  in  Mny-D-87269  and  MIL-M-87268  is  under  preparation  by  the  DOD- 
esrtablished  Hi-Service  EETM  Working  Groip.  This  Specification  or  Ilandbook  is  scheduled  for 
conpletion  in  the  third  quarter  of  FY  1994. 


6.2.1  Recommendation  for  Navy  Use 

The  Navy  should  continue  to  assess  the  effectiveness  of  available  commercial  Portable  Electronic 
Display  Devices.  An  in-house  Navy  development  does  not  appear  to  be  needed.  Industry  should  be 
encouraged  to  develop  rugged,  capable  PEDDs  and  associated  operating  and  presentation  software  uqiable 
of  operating  on  a  wide  range  of  suitable  devices.  The  present  processing  and  display  requirements  for  a 
shipboard  or  flightline  PMAl’EDD  do  not  seem  beyond  the  current  industrial  state  of  the  art,  and  with 
the  addition  of  environmental  capabilities  (e.g.,  tenperature  extremes,  mggedness,  salt,  and  EM1)> 
Commercial-Off-Ihe-Shelf  (COTS)  devices  should  provide  adequate  basic  systems  for  Navy  PMA  use. 
Existing  COTS  software  for  such  devices  is  generally  considered  unsatisfactory  and  would  require  rework 
or  conplete  development  in  many  particulars. 


6.3  IMPROVEMENT  OF  GRAPfflCS  PRESENTATION 

Work  sliould  be  undertaken  to  provide  needed  inprovements  in  grphics  diplays  (designed  for 
bench-mounted  Electronic  Diplay  Devices  and  for  PMAs)  and  in  the  gnphics  themselves  (e.g., 
elimination  of  stairstepping),  in  the  speed  of  handling  graphics  (e.g.,  providing  the  "next"  screen  in  less 
than  one  second),  and  in  conprehensible  presortation  of  the  type  of  information  which  is  usually  provided 
to  teclmicians  on  large-scale  paper  drawings.  As  shown  by  this  Test,  currently  epressed  requirements 
for  scrolling  are  difficult  to  inplement  satisfactorily,  and  should  be  eliminated  if  effective  alternative 
diplay  techniques  can  be  found. 
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6.4  ASSESSMENT  OF  REDUCTION  IN  TRAINING  REQUIREMENTS  WITH  lETM  USE 

This  Test  (and  others)  have  shown  the  effectiveness  of  hi|^y  proceduralized  fault-isolation  and 
corrective-maintenance  procedures  in  significantly  iiiproving  performance  of  ine?q)erienced  technicians. 
Instruction  required  for  use  of  ar  lEIMPMA  combination  itself  is  minimal.  The  interaction  between  the 
need  for  schoolhouse  and  on-board  training  and  the  use  of  lElTds  should  be  carefully  assessed  with  an 
eye  to  significant  reduction  in  formal  training  requirements,  and  to  define  the  level  to  which  training 
information  should  be  incorporated  into  the  lETM  itself. 


6.5  INTERACTION  OF  DilTMs  WITH  OTHER  MAT^^CE-RELATED  FUNCTIONS 

An  lEIM/PMA  combination  has  been  shown  to  be  most  effective  in  sTeamlining  accoirplishment 
of  interactive  functions  involving  maintenance  and  other  logistics  processes  (e.g.,  parts  ordering  and 
maintenance  reporting).  It  is  recommended  that  effort  be  carried  out  or  increased  to  define  the  extent  to 
which  other  maintenance  and  maintenance-related  functions  should  be  integrated  with  lEIMs  (m  addition 
to  those  which  were  demonstrated  in  this  Test).  The  achievement  of  cornplete  Technical  Information 
integration  throughout  the  entire  maintenance  process,  and  provision  of  automated  interfacing  with 
associated  management  systems  such  as  NALCOMIS,  offer  great  promise  of  increased  efficiency  (in  terms 
of  reduced  time  and  costs)  all  along  the  logisdes  chaia  Such  an  integration  of  ail  logistics-siqiport 
Technical  Information  is  the  basis  of  IMIS  as  well  as  of  the  Navy  AMEDD  (Airciaff  Maintenance 
Integrated  Diagnostics  Demonstration)  concept 

6.6  IMPROVEMENT  IN  BITE  COUPLING  WITH  PMA 

Based  on  effective  interaction  between  the  PMA  and  on-aircraft  BITE  throu^  use  of  the  1SS3  bus, 
as  demonstrated  in  this  Test,  future  on-aircraft  BITE  installadors,  design  of  DSUs  for  fliture  aircraft  (and 
for  other  weapon  systems),  aivd  other  test-equipment  considerations  should  take  into  account  the  possibility 
of  direct  coiqsling  to,  and  interactive  information  exchange  with,  a  PMA  or  other  lEIM  display  system. 
Mechanisms  for  such  coi^ling  (e.g,,  cables,  plugs)  need  wotk  to  minimize  interfeience  with  other 
procedures  or  cumbersome  arrangenents. 
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6.7  REVIEW  OF  DYNAMIC  DIAGNOSTICS 


The  concept  of  "dynamic  diagnostics"  (i.e.,  autoniated  Artificial-Intelligence  type  application  of 
information  accumulated  in  the  Display  Device  itself  during  the  maintenance  history  of  a  ^veapon  system, 
used  to  improve  Fault-Isolation  procedures  on  a  case-by-case  basis)  needs  additional  development  effort 
and  laboratoiy  testing  prior  to  further  field  testing.  Also,  guidance  should  be  prepared  to  provide 
Contractors  with  trade-off  considerations  to  permit  choices  between  TI  prepared  under  full  control  of  a 
human  author  and  Tl  fully  automated  by  Delivery  Device  algorithms  based  on  direct  interaction  with  the 
lETMDB. 


6.8  CONTINUED  IMPROVEMENT  IN  ELECTRONIC  DISPLAY  OF  TI 

It  is  clear  fiom  this  Test  (and  from  previous  operational  tests)  that  further  work  needs  to  be  done  on 
optimization  of  the  display  of  Technical  Information  on  luminous  screens  of  all  types.  This  effort  should 
be  based  on  work  done  to  in^ve  graphics  (see  section  6.3)  and  on  work  done  to  improve  PMA  display 
epability  (see  section  6.2),  but  will  also  require  Anther  Human  Factors  effort  of  the  type  described  in 
ref  11.  Results  of  such  efforts  should  be  standardized  and  incorporated  into  MIL-M-87268. 
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by  The  Tri-Service  Working  Groi^  for  Interactive  Electronic  Technical  Manuals.) 
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ABBREVIATIONS  AND  ACRONYMS 


1553 

3M 

AF 

AUHRGO 

AMIDD 

ARI 

ATE 

BIT 

BITE 

BUN 

CALS 

CDI 

COM 

CDNSWC 

CGM 

CMAS 

CND 

COTS 

DDI 


Aircraft  Maintenance  Data  Bus  Standard  1553  on  USAF  F>16,  and  Navy/USMC  F/A*18 
Aircraft 

Maintenance  and  Material  Management 
Air  Force 

Amistrong  Laboratory/Air  Force  Banan  Resources  Directorate,  Wri^t-Patterson  Air 
Force  Base,  Dayton,  Oiio 

Aircraft  Muntenance  Integrated  Diagnostics  Demonstration 

Army  Research  Institute 

Automated  Test  Equipment 

Built'in  Test 

Built-in  Test  Equipment 

BIT  Logic  Inspection  (number) 

Conpiter-aided  Acquisition  and  Logistics  Siqjport 

Collateral  Duty  Inspector 

Content  Data  Model;  Basis  of  lEIMDB 

Carderock  Division,  Naval  Sur&ce  Warfare  Center,  Bethesda,  MD 

Computer  Qnqi^cs  Metafile 

Conqxiter-based  Mcdntenance  Aids  System:  lYedecessor  of  the  FMA  of  this  Test 
Can  Not  Diqjlicate 
Comnercial-Ofif-lhe-Shelf  (product) 

Digital  Display  Indicator 
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Abbreviations  and  Acronyms^  cont’d 


DM  Diagnostic  Module 

DSU  Data  Storage  Unit 

EDS  Electronic  Display  System 

FOC  F/A>18  Fli^  Control  Con^puter 

FCCA  FCC  "A" 

FOCB  FCC  "B" 

FCS  F/A-18  Fligiit  Control  System 

FRM  Fault  Reporting  Manual 

GCSFUI  Genercd  Content,  Style,  Forma  and  User  Interaction  Requirements  Specification 
(now  issued  as  AttL’HM7268,  dated  20  Nov  1992) 

GSE  Ground  Sifiport  Equipment 

lOS  Hunan  Confiuter  Interface  Specification  (See  ref.  1.) 

HF  Human  Factors 

IBIT  Initiated  Built-in  Test 

lETM  Interactive  Electronic  Technical  Mmual 

lETMDB  Interactive  Electronic  Technical  Manual  Data  Base  (Specification  now  issued  as 
MtL-M7269,  dated  20  Nov  1992) 

IMIS  Integrated  Maintenanoe  Information  System 

AKIAIR  McDonnell  Douglas  Aircraft  Con^y 

MBIT  Maintenance  Built-in  Test 

MC  Maintenance  Control 

MDAS  Maintenanoe  aitd  Diagnostic  Aiding  System 
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Abbreviations  and  Acronyms,  cont'd 


MIPS 

MOS6337 

MOS6317 

MSP 

MIBF 

MCAS 

NALCOMIS 

Ncx:&asc 

NPRDC 

MIPS 

INSWC 

PBrr 

PEAM 

FEB 

PEDD 

FCMAS 

FMA 

PS 

BAD 

R&M 

RMTS 


Million  Instructions  Per  Second 

USMC  Militaiy  Occi^tional  Specialty  -  Electrician 

USMC  Military  Occupational  Specialty  •  Coininunication/Navigatioii/Radar 

Mdntenance  Status  Panel 

Mean  Time  Between  Failures 

Marine  Corps  Air  Station 

Naval  Logistics  Command  Management  Information  System 

Naval  Command  Control  and  Ocean  Surveillance  Center,  San  Diego,  CA 

Navy  Personnel  Research  and  Development  Center 

Navy  Technical  Information  Presentation  System 

Naval  Surface  Warfare  Center 

Periodic  BuilMn  Test 

Personal  Electronic  Aid  for  Maintenance 

Pre-Expanded  Bin 

Portable  Eectronic  Delivery  Device  (a  Navy  term  equivalent  to  PMA) 
Portable  Conplte^bBsed  Maintenance  Aid  System 
Portable  Maintenance  Aid  (IMIS  term  equivalent  to  PEDD) 

PresentatiQn  System 
Research  and  Development 
Reliability  and  Maintainability 
Rudder  Manual  Trim  System 
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Abbreviations  and  Acronyms,  cont'd 


R&R  Remove  and  Replace 

RTOK  Retest  OK 

SATD  Strike  Aircraft  Test  E)irectorate,  Patuxent  River  Naval  Air  Station 
SEI  Systems  Exploration,  IncDtporated 

SGML  Standard  Generalized  MwIoq)  Language 

SMA  Scientific  Management  Associates,  Inc. 

SME  Subject  Matter  Ejqsert 

SRL  Systems  Research  Laboratory 

T&E  Test  and  Evaluation 

n  Technical  Informadon  (Paper-based  ot  Electronic) 

TIDER  Technical  Information  Deficiency  and  Evaluation  Report 

TG  Test  Group  (procedure) 

VUDS/MAP'  'Visual  Information  Display  System/N&intenance  Action  Form  (NAVAIR  maintenance- 
control  and  reporting  form) 

WC  Work  Center 

WP  Work  Package 

>VRA  Weapon  Replaceable  Assembly 
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APPENDIX  A 


A  DetaUed  Presentation 
of 

Technicians*  Debriefing  Comments  on  the  lETM 
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INTRODIJCnON 


Technicians'  opinions  conceming  the  lEIMPMA  combination  ^vere  solicited  at  the  following  stages 
of  the  Test: 

□  After  each  of  their  six  troubleshooting  pofonnances. 

□  As  part  of  the  Questionnaire,  they  were  asked  to  conplete  after  all  fault  exercises. 

□  During  a  Structured  Interview  following  conqjletion  of  all  fault  exercises. 


Subject  Technicians  were  encouraged  to  identify  the  positive  features  of  the  lETM/PMA  and  to 
provide  any  suggestions  for  inpvvetnent.  The  two  subsections  below  summarize  the  opinions  reported 
by  the  Technicians;  first,  with  respect  to  lEIM  features  which  they  thought  effective  and  beneficial  and, 
second,  with  respect  to  the  features  which  they  believed  needed  inpoved 


1.  Effective  Features  of  the  lEIM/PMA 


a 


Integration  of  Information.  Topical  of  the  several  comments  on  this  feature  was:  "Everything  you 
need  is  ri^t  there;  you  don't  have  to  page  throui^  a  bunch  of  books".  The  consolidation  the 
Technicians  were  referring  to  includes  the  side  by  side  presentation  of  text  and  sqjporting 
graphics,  the  linking  of  cross  reference  or  branching  procedures  and  the  ability  to  call  locator 
graphics,  w^ch  are  accessed  via  Required  Conditions  or  a  soft  key.  One  negative  aspect  of  this 
feature  concerned  allocation  by  authors  of  FMA  instructions  for  locators  among  the  three  access 
techniques;  e.g. ,  some  Technicians  thought  that  too  many  locators  had  been  made  an  integral  part 
of  the  text-gri^hic  instructions,  which  slowed  their  performance. 

b.  Automated  Functions.  Technicians  were  enthusiastic  about  the  automation  of  what  had  been 
manual  and  sotnetimes  error  -prone  tasks.  The  automated  versions  of  the  tasks  consisted  of: 


(1)  VIDS/MAF  conviction  (especially  the  Parts-Ordering  section). 

(2)  Memory  Inspect. 

(3)  Control  of  the  PCS  BIT  via  the  1553  multiplex  bus. 

c.  Frmr  Prevention/Reduction.  This  lETM/PMA  benefit  was  identified  by  Technicians  during 
debriefings  and  quantified  by  the  data  collected  during  the  test.  An  exanple  of  the  types  of 
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errors  which  are  preventable  by  lETMs  but  allowed  by  paper  TMs  is  the  back-and-forth  method 
of  doing  maintenance  in  accordance  with  any  information  source.  When  using  paper  TMs,  the 
open  book  usually  shows  facing  pages.  After  conpleting  a  step  at  the  aircraft,  the  technician 
returns  to  the  book  to  regain  his  place  in  the  instructional  sequence.  Occasionally,  the  technician 
resumes  the  instructional  sequence  at  a  wrong  place  (for  exanple,  on  the  right-hand  page  instead 
of  left-hand  page,  or  by  dropping  a  row  when  moving  to  the  right  to  get  the  "Go  To"  instruction). 
Technicians  in  this  test  committed  both  error  types  leading  to  failures  to  solve  the  problem 

d  PMA  Portability.  Technicians  pointed  out  that  the  PMA's  relative  portability  constituted  an 
advantage  as  conpired  to  the  numerous  TMs  they  were  required  to  carry  to  provide  an 
equivalent  amount  of  information.  In  addition,  Technicians  assessed  the  PMA's  size  and  weight 
a«  acceptable. 

e.  Selection  of  Information  for  Display.  Technicians  approved  ofthe  PMA's  capability  to  select  for 
display  only  that  information  relevant  to  the  Technician's  assigned  aircraft;  e.g.,  the  Technician 
no  longer  needed  to  review  the  paper  TMs  blocks  of  tail  numbers  to  find  the  block  and  the 
associated  procedure  which  applies  to  his  aiicraft.  This  lEIM  capability  eliminates  what  has 
been  an  error-prone  task.  Using  the  pt^  IM,  more  than  one  Technician  followed  a  wrong  set 
of  procedures  through  making  the  wrong  "EfTeedvity"  choice. 


2.  Suggestions  for  Fdilher  Improving  the  lElM 

One  of  the  Structured  Interview  questions  asked  Technicians  to  note  other  maintenance  or 
maintenance-related  functions  that  might  be  sqjported  by  flying  the  lEIM  automation  approach. 
Althou^  the  services  are  already  pursuing  many  of  the  suggested  capabilities,  they  are  reported  here  as 
a  field  endorsement  of  such  programs.  The  following  is  a  summary  of  the  Technicians'  suggestions. 

a.  Link  to  Maintenance  Control  (MQ.  The  purpose  of  the  link  would  be  to  provide  MC  with  real 
time  job  status  information  In  fact,  a  much  broader  link  is  under  developnent;  i.e.,  dun^)  of  the 
DSU,  pilot/crew  debriefing. 

b.  Next  Day's  Flight  Schedule.  One  Technician  suggested  that  incorporation  of  iqxoming  flight 
schedules  by  Maintenance  Shops  would  allow  Technicians  to  target  their  efforts  more  effectively, 
e.g.,  concentrate  on  the  misrion  equipment  needed  to  siqjport  the  schedule. 

c.  PMA/DSU  Interface.  This  suggestion  is  directed  at  obtaining  directly  &om  the  DSU 
maintenance-relevant  information  which  is  not  now  available  to  the  Technicians;  e.g.,  deeper 
Fault  Isolation  (to  a  conponent  within  a  We^ns  Replaceable  Assembly  (WRA)  as  compared 
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with  one  or  more  WRAs),  and  flight  conditions  which  prevailed  at  fault  detectioa  This  feature 
is  also  under  active  development. 

d.  Adding  Training  Content  to  the  lETM  The  Technician  making  this  suggestion  perceived  the 
lETM  as  having  the  potential  to  siqsport  On-the-Job-Training  for  new  squadron  personnel.  This 
capability  is  also  under  development  by  the  Services. 

e.  Work  Planning.  Some  of  the  Technicians  reported  that  their  normal  troublesho-  ting  starts  with 
a  planning  session  to  "scope”  out  the  {robiem,  and  they  believed  that  the  P;  inhibited  this 
phase  of  their  work.  They  suggested  that  a  browse  mode  mi^t  be  included  in  the  PMA  to 
facilitate  this  process.  They  believe,  also,  that  the  troubleshooting  diagrams  (the  coded  block 
diagrams)  arKl  the  schematics  have  potential  to  meet  this  need 

f  PMA's  Response  Time.  The  eltqsed  time  between  pressing  "NEXT'  and  the  appearance  of 
follow-on  screen  was  deemed  too  long  by  some  Technicians,  especially  when  a  complex  graphic 
was  part  of  the  follow-on  sctoea  In  sonne  instances,  the  response  time  approached  10  seconds. 
One  senior  Technician  con^lained  that  this  was  especially  frustrating  when  after  the  gr^hic  of 
the  follow-on  screen  {qipeared,  he  found  he  didn't  need  it.  He  would  have  preferred  a  "checklist- 
type"  presentation;  e.g.,  brief  textual  statements  of  steps  with  siqjporting  gr{q>hics  available  via 
a  soft  function  key,  (This  objection  has  been  repeatedly  made  in  lETM  field  tests.  Gr^ihics 
presentation  software  is  still  unsatisfactory  in  this  regard  Enphasis  must  be  added  to  efforts  to 
in^ve  this  aqnbility.) 

g.  Prff.F.xpended  Parts  As  already  noted,  the  automated  Parts-Qrdering  process  received  strong 
sipport  fiorn  the  Technicians;  all  used  it  and  all  endorsed  it.  However,  the  process,  as  included 
in  the  PMA  for  this  test,  did  not  allow  the  Technicians  to  follow  their  normal  practice  of 
checking  a  local  Pre  Extended  Bin  (PEB)  for  smaller  parts,  as  conqsared  to  conq>leting  the  more 
time-consuming,  formal  ordering  process.  This  situation  occurred  in  the  two  Relay  Tests;  relays 
are  parts  sometimes  included  in  the  PEB.  Technicians  commenting  on  this  aspect  of  the  PMA 
suggested  that  irKluding  a  PEB  feature  would  be  an  additional  benefit. 

h.  Additional  Locator  Information  on  Wiring  Diagrams.  The  lETMs  used  a  novel  type  of  wiring 
diagram  whose  overall  circuitry  is  limited  to  that  which  occurs  between  two  components  of 
interest,  and  whose  circuitry  detail  was  far  less  than  conventional  schematics.  The  reaction  to 
these  diagrams  was  mostly  favorable  but  one  consistent  request  was  that  locator  information  be 
included  between  end  points  of  the  diagram,  i.e.,  add  locations  of  doors,  bulkheads  and  panels 
between  the  end  points.  The  rationale  for  this  request  was  that  the  additional  information  would 
allow  Technicians  to  save  time  b>'  using  the  half-split  technique.  In  addition,  a  request  was  made 
to  include  test-tolerance  information  where  appropriate. 
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i.  Abbreviated  Wiring  Diagrams  vs  Schematics.  Technicians'  reaction  to  the  abbreviated  wiring 
diagrams  was  mixed,  some  considering  that  the  detail  was  adequate,  others  that  the  detail  was 
not  sufficient.  The  latter  Technicians  believed  that  the  abbreviated  wiring  diagrams  were 
adequate  for  the  faults  used  during  the  test,  but  that  they  would  be  inadequate  for  more  con^lex 
faults.  In  addition,  they  believed  that  full  schematics  would  provide  them  with  an  overview  of 
the  scope  of  the  fault,  information  not  available  elsewhere  in  the  lEIM  They  believed  that 
without  schematic  information,  they  had  little  basis  for  following  the  logic  of  the  lETMs 
procedural  sequence.  In  essence,  they  were  asking:  How  can  technicians  assess  "Recommended 
Tests"  and  "Ranked  Actions"  if  they  have  no  means  for  understanding  the  weapon  system's 
hardware?  They  believed  that  some  way  needs  to  be  found  to  present  fuller  schematics  on  small 
screens.  Techniques  suggested  for  accomplishing  this  included  "select  a  section  and  zoom"  and 
"scrolling". 

j.  Level  of  Detail.  Comments  both  pro  and  con  were  offered  on  the  explicitness  of  the  DilM 
instructions.  Some  respondents  believed  that  gearing  the  instructional  e:q>licitness  to  the  novice 
was  a  pood  feature  that  would  cut  down  on  human  error;  others  believed  it  was  too  detailed  and 
slowed  the  performance  of  the  more  experienced  Technicians.  These  comments  suggest  that  a 
need  exists  for  further  research  on  balancing  the  "Expert  •  Novice"  presentations  of  the  two  track 
system. 

k.  Inadvertent  Key  Activation.  Use  ofthePMA  in  perfonning  the  six  Fault-Isolation  tests  involved 
considerable  movement  of  the  PMA;  e.g.,  to  the  aircraft,  fiom  the  ground  to  the  cockpit,  and 
back.  Technicians  observed  that  during  these  moves  it  was  easy  to  press  a  key  inadvertently, 
resulting  in  a  new  and  unwanted  screea  One  Technician  suggested  the  inclusion  of  a  lockout 
key  to  prevent  these  disr^Xivc  rxxxsrences. 

l.  Screen  Glare.  Technicians  commented  that  even  at  angles  of  view  near  90°,  it  was  occasionally 
difficult  to  make  out  screen  content  because  of  glare.  These  same  Technicians  were  quick  to 
point  out  that  the  backlit  feature  was  a  great  countermeasure  for  this  problem,  yet  they  realized 
that  this  design  feature  involves  a  trade-off  because  of  the  extra  power  needed  for  the  backlit 
mode. 

m.  PMA  Awkward  in  Cockpit.  Much  of  the  troubleshooting  work  is  performed  in  the  cockpit  and 
some  portion  of  this  involves  movement  of  the  flight-control  stick.  Technicians  corr^loined  that 
the  PMA  is  an  inpediment  to  the  stick  movement. 

n.  1SS3  Bus  Interface.  The  overall  opinion  as  to  the  1SS3  link  to  the  aircraft  was  positive,  but 
improvement  was  requested  for  several  aspects. 
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(1)  The  hcxjkiqj  required  disabling  the  mission  computers  which  in  turn  disabled  the 
Digital  Display  Indicators  (DDls).  This  criticism  stated  that  the  conputers  are  needed 
to  collect  Engine  Life  data,  and  Technicians  preferred  to  have  the  DDIs  active. 
NOTE;  Taking  the  DDIs  and  computers  off-line  was  an  e?q)ediency  for  the  test  and 
would  not  be  done  for  operational  use. 

(2)  The  Nosewheel  Well  location  of  the  interface  plug  precludes  any  maintenance  action 
that  requires  opening  and  closing  of  the  Nose^\heeI  doors. 

(3)  The  1553  umbilical  cord  deterred  PMA  portability. 

(4)  The  feed  fiom  the  data  bus  to  the  PMA  nvbs  presented  on  two  screens  \\hich  had  to 
be  conpared  However,  the  screens  to  be  compared  were  not  adjacent  to  each  other, 
making  the  conparison  more  difficult  than  necessary. 

0,  Watch  Icons.  It  was  observed  that  a  standard  location  and  a  more  attention-getting  appearance 
are  desirable  for  the  Watch  icon  (the  icon  indicating  that  the  conputer  was  working). 

p.  Cursor  Expectancies.  A  conplaint  was  made  that  the  cursor  movement  violated  ejqsectations, 
e.g.,  a  press  of  the  ri^t  arrow  key  did  not  always  result  in  a  movement  to  the  ri^. 

q.  Series  of  Checks.  The  F/A-18  troubleshooting  {siocedures  include  many  instances  in  y/tnch  a 
series  of  checks  must  be  made;  e.g.,  a  set  of  continuity  checks.  In  the  lETM  treatment,  when 
the  Technician  entered  his  first  No  Go,  the  device  branched  to  the  next  step  (rather  than 
conpleting  the  remainder  of  the  checks).  This  procedure  was  considered  undesiiabie  because, 
after  resolving  the  fault  related  to  the  first  No  Go,  the  System  Health  check  might  fail  and  lead 
back  to  the  inconplete  series.  The  commenting  Technician  believed  that  the  lEIM  ought  to 
allow  conpletion  of  the  set  of  checks. 

r.  Limited  Access  to  Important  Information.  Technicians  complained  that  some  inportant 
information,  notably  Initial  Set  Up  and  Fault  Verification,  was  available  only  via  the  \TDS/MAF 
(Part  I),  and  further,  it  was  not  possible  to  backtrack  into  the  VIDS/MAF  from  a  downstream 
positioa 

s.  Cursor  Control.  The  PMA  offered  both  a  thumb  knob  and  the  arrow  keys  as  means  of 
controlling  the  cursor.  The  Technicians  favored  the  arrow  keys  over  the  thumb  knob.  No 
reasons  were  given. 

t.  Number  Keys.  Technicians  favored  the  number  sequence  1  to  0  over  the  0  to  9  sequence. 
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a  Operational  Realism.  The  Ground-Support  Equipment  (GSE)-intensive  configuration  of  the  Test 
setiq}  was  criticized  as  being  in  conflia  with  the  operational  practice  of  using  the  aircrafts 
hydraulic  pressure  rather  than  the  external  hydraulic  generators  used  in  the  test.  The  criticism 
is  legitimate,  but  the  external  hydraulic  generators  were  a  test  e^qiedient  and  not  intended  as  an 
operational  measure.  Other  comments  about  the  lack  of  realism  in  the  PMA  included  the  way 
in  which  VIDS/MAFs  were  prepared  for  the  multiple  faults  ("Maintenance  Control  doesn't  do 
them  that  way"). 

V.  Unfriendly  Abort  and  Restart.  One  Technician  noted  that  the  PMA  does  not  offer  a  user-friendly 
way  to  abort  and  restart  a  process  such  as  a  test  sequence. 

w.  Collateral  Duty  Inspector  fCDfi  Sign  Off.  A  Technician  suggested  that  lETM  should  not 
advance  to  the  next  segment  of  the  procedure  until  the  Collateral  Duty  Inspector  signs  off  at 
designated  point(s). 

X.  Confusing  Similarities  among  Required  Conditions.  Follow-On.  and  Closing  Actions.  The 
conilision  among  these  lETM  presentations  was  based  on  their  visual  similarity  and  the 
fqjpearance  of  repetition.  As  an  exan^le,  Technicians  conplained  that  they  were  being  directed 
to  reconnect  plugs  only  to  find  that  a  later  screen  called  for  disconnecting  the  same  plug.  These 
procedures  should  be  reviewed  for  possible  consolidation 

y.  Diagnostic  Block  Diagram  Comments  on  these  diagrams  were  pasitive  (e.g,  they  narrowed 
down  the  possibilities,  showed  the  interrelationships)  but  contained  some  negative  inpessions 
(e.g.,  marginally  acceptable,  somewhat  confusing,  good  but  mainly  applicable  to  more  coiTq)lex 
problems,  did  not  contribute  over  and  above  troubleshooting  instructions).  This  mixed  review 
indicates  a  need  to  define  more  precise  uses  for  this  capability;  e.g.,  redo  the  diagnostic-block 
diagrams  to  serve  as  an  overview  of  the  &ult,  a  need  Technicians  were  not  able  to  meet  by  using 
existing  lElM  content. 

z.  Fragility  of  the  PMA  The  need  to  treat  the  PMA  gently  was  identified  as  a  characteristic  which 
needs  design  attendoa  PMA  ruggedization  is  an  established  requirement  for  which  draft  Military 
Specifications  have  been  established  and  appropriate  development  efforts  are  underway. 

aa.  Capability  to  Withstand  Harsh  Environmental  Conditions.  Saudi  Arabia  and  Somalia  were 
mentioned  as  typical  operating  environments  for  the  USMC,  and  of  course  the  PMA  would  have 
to  be  able  to  function  under  environmental  conditions  encountered  in  those  countries.  Such 
environmental  requirements  are  well  known  and  PMA-design  improvements  are  under 
development. 
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ab.  Detter  Labelling  of  Soft  Function  Kevs.  The  specific  complaints  were  not  stated  but  a  review 
of  the  key  labels  would  be  profitable- 

ac.  Battery  Life  and  Recharpe.  The  operating  life  of  newer  batteries  is  considerably  betta  than  the 
batteries  used  in  the  test.  However,  the  life  and  recharge  constraint  is  a  limitation  and  is  being 
worked  on. 

ad  Alert  Cue  for  Long  Series.  The  lETM  presents  long  series  of  tests  (e.g.,  Left  Stab  Intermittent 
Wiring  tests)  on  three  or  more  screens.  The  technician  can  waste  time  by  reading  the  first 
instruction,  going  to  the  aircraft  to  carry  out  the  test,  returning  to  the  PMA  to  enter  the  result  and 
then,  ipin  advancing  to  the  next  screot,  finding  that  he  has  toretum  to  the  previous  test  location 
The  technician  suggested  including  a  prompt  at  the  outset  of  these  series  to  minimize  any  such 
back-and-forth  time,  e.g.,  look  ahead  and  jot  down  the  pin  numbers,  or  carry  the  PMA.  with  him 
if  location  of  tests  is  within  the  cord  radius. 

ae.  Time  Spent  So  Far  Against  Normal  Fractional  or  Total  Time.  One  technician  suggested  that  the 
PMA  present  some  means  of  showing  average  task-conpletion  time,  with  a  running  indication 
of  how  far  into  this  average  time  the  work  had  progressed  (e.g..  a  bar  chart  showing  25%,  50%, 
75%,  100%). 

af.  Clarification  of  the  Highlight  and  Select  Functions.  In  a  typical  use  of  these  PMA  keys,  a 
technician  would  move  the  cursor  ovo'  the  desired  item  (this  action  hi^i^ts  the  item),  then 
press  the  SELECT  key  and  finally  press  akey  to  Implement  the  pertinent  action  Apparently, 
other  PMA  sequences  (or  other  conputere}q)eriences)  led  techrticians  erroneously  to  believe  that 
pressing  only  NEXT  after  hi^i^ting  an  item  in^lemented  that  item's  action.  This  confiislon 
led  to  wasted  time,  some  unexpected  screen  advances  and  occasionally  an  erroneous  iiput  to  the 
PMA  For  exanple,  in  a  list  of  test-result  options  with  one  defaulted,  the  technician  moves 
cursor  over  a  non  default  value,  resulting  in  its  being  highii^ted  He  then  presses  NEXT 
intending  this  action  to  enter  the  highli^ted  item  into  the  PMA  In  fact,  without  first  pressing 
SELECT,  pressing  NE7iT  will  enter  the  default  item  (instead  of  the  highlighted  item)  into  the 
PMA  The  faulty  sequence  results  in  sonous  deNdations  fiom  the  desired  diagnostic  path. 

ag.  T  j^M-Than-Full-Use  of  Function  Keys.  Certain  PMA  screens  offer  the  technician  more  than  one 
way  to  interact  with  or  manipulate  the  screen:  hard  keys,  highli^t  and  select,  soft  key,  NEXT 
to  a  default.  Most  technicians  speared  to  learn  and  then  go  with  one  technique  over  the 
alternatives  even  though  it  might  have  been  more  efficient  and  user  friendly  to  use  techniques 
best  suited  to  the  situation  at  hand,  e.g.,  NEXT  for  defaulted  option;  soft  key  for  choosing  special 
options. 
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BIOGRAPHIC  DATA  SHEET 


Name: _  Participant  #:, 

Rank/Rate/Job  Title: _  NEC: _ 

Time  in  Service: _  Time  in  NEC: _ 

Command/Activity  Name  &  Location: _ 

Dept.  Title/Code: _ 

Are  you  right-handed? _ Yes  _ No 

Years/Months  at  this  Command/Activity: _ 

Navy  Schools  Attended  (Provide  dates): 


F/A-18  Maintenance  Experience  (Years/Months): _ 

Note:  List  specific  f^/A-lB  systems/subsystems  you  have  experience 
on,  i.e.  the  flight  control  system. 

Specify  percentages: 

Hands-on: _  Instructor: _  Other  (Specify): _ 

Other  Aircraft  Maintenance  Experience  (Specify): _ 


Do  you  have  any  computer  programming  training  experience? 
(Piease  specify,  e.g.,  octal,  binary,  digital  fundamentals,  etc.),  e.g,, 
Course  taken,  language,  familiarity,  etc. 


Please  summarize  the  computer  type  and  applications  you  are 
familiar  with: _ _ _ 


IHS  PAC3E  108  INIEraOiALLY  LEFT  BLANK 
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Pariici^iant 


PERFORMANCE  OBSERVATION  FORM 


Problem  # 

Presentation  Method 

Date  of  Test 

Initials  of 
Observer(s) 

PMA 

Manual 

SECTION  1:  PRELIMINARY  ACTIVITIES 


iNoie:  Eniries  to  this  section  should  be  made  immediately'  after  the  VIDS/MAF  is  first | 
resented  to  the  test  paflicipant. 


1,  START  TIME: 


2.  After  obtaining  the  VIS/MAF,  list  any  problems  (e.g.,  finding  the  technical  manuals, elc.)  you  detect  in 
gathering  the  technical  information  and  tool  kit  prior  to  exiting  the  work  center,  Please  note  any 
problems  in  accessing  the  VIDS/MAF  via  the  PMA,  e  g,,  does  the  technician  make  (or  appear  to  make) 
inappropriate  button  presses  on  the  PMA. 


I 

I 

I 

I 

I 

I 

I 

B 


3.  How  much  lime  did  the  technician  spend  reviewing  the  technical  infonnation? 

_ START  _ STOP  _ NONE 

4.  STOP  TIME:  _ 

SECTION  2:  SET-UP  AT  AIRCRAFT 

1 .  START  TIME _ 

2.  Describe  set-up  problems  for  placing  PMA  or  WP  at  the  aircraft,  [For  PMA,  include  connection  to  the 
1553  bus.] 


3 .  Note  any  problem  "verifying*  electrical  and  hydraulic  hook-ups. 


4.  Detail  steps  technician  took  to  verify  fault  symptoms  documented  on  VIDS/MAF  prior  to  initiating 
troubleshooting,  i.e,,  describe  the  process  for  Initiating  the  BIT  (PMA  should  be  straight  torward  with 
the  PMA  controls;  WP  will  require  knowledge  or  technical  information  referencing  to  run  the 
appropriate  BIT). 


5.  STOP  TIME: 
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SECTION  3:  TROUBLESHOOTING  ACTIVITIES 


Note:  Entries  to  this  section  should  be  made  immediately  after  the  fault  symptoms  have 
been  verified  through  the  approprlafe  BIT, 


I 

I 

t 


1.  START  TIME:  _ 

2.  In  the  spaces  below,  list  ar>d  briefly  describe  all  fault  detection  and  isolation  tests  in  the  order  each  was 
performed:  and,  note  whether  the  test  was  valid  or  invalid.  Include  applicable  technical  information 
consutted  for  each  lest  performed,  then  briefly  describe  the  results  of  each  lest  under  the  heading: 
‘Test  Outoome.'  Continue  on  additional  sheets  as  necessary. 


Description  of  Fault  Detection/ 

Isolation  Action(s)  Performed 

Tech,  Info. 
Checked 

Steo  No. 

Note  whether  it  was  a  valid  or  invalid  test 

VoL/Section 

Test  Oulcome 

3.  List  each  time  the  technician  concludes  that  he/she  has  found  the  failed  component,  and,  the 
outcome,  i.e.,  success  or  failure. 

I - ^ - 


4.  STOP  TIME; _ 

113 


Participant  # 


SECTION  4;  REMOVE  AND  REPLACE 


1,  START  TIME;  _ 

2.  Describe  any  problems  the  technician  has  with  accessing  the  correct  *Remove  ar>d  Replace'  procedure 


5,  STOP  TIME: 


SECTION  5;  REPLACEMENT  PARTS  IDENTIFICATION  &  REQUISITIONING 

”te:  Entries  to  ihis  section  should  be  made  immediately  alter  the  lest  participant  has! 
_^^_^^^^^^^^^_isolaiedUj^^£C^njaMuncijor^ojhe^jti^j2eagon^re£|acea^ 

1,  START  TIME:  _ 

2  Alter  identilying  the  correct  'Remove  and  Replace*  procedure,  detail  all  the  steps  the  technician 
takes  to  access  pans  information  and  identify  the  faulty  V/RA.  Note  all  problems. 

3  Note  that  all  entries  to  the  supply  requisition  (i.e.,  for  ordering  a  replacer  »ni  WRA)  are  complete  and 
correct.  Note  all  problems. 


Panicipant 


SECTION  6:  REQUISITIONING  PROCEDURES 


1,  START  TIME:  _ 

2.  After  requisitioning  the  pan,  detail  all  the  steps  the  technician  takes  to  complete  the  required  paper 
work.  Note  all  problems. 


3.  STOP  TIME 


SECTION  7:  INSTALLATION  AND  CHECKOUT 

I 

1,  START  TIME;  _ 


2.  Although  simulated,  the  technician  will  be  ‘given*  the  replacement  pan  and  asked  to  ‘install*  H. 
Describe  any  problems  the  technician  has  in  accesssing  the  correct  'Remove  and  Replace* 
procedure. 


3.  After  the  ‘new*  component  has  been  installed,  detail  the  verificalion  process  the  technican  follows  to 
ensure  the  aircraft  is  operational. 


4.  STOP  TIME: 
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SECTION  8:  VIDS/MAF  COMPLETION 


Hole:  Entries  to  this  section  should  be  made  immediately  after  the  test  participant 

received  instructions  from  the  test  administrator  to  fill  out  the  VIDS/MAF  lor  the 
troubleshooting  probiem  just  completed.  It  is  anticipated  that  this  activity  will  be 


1.  START  TIME;  _ 

2.  Was  correct  technical  information  accessed  and  used  when  completing  entries  to  the  VIDS/MAF? 

_ YES  _ NO. 

II  'NO,"  list  discrepancies  obse  n  the  spaces  below; 


3,  Were  all  applicable  data  entries  to  the  VIOS.'MAF  complete  and  correct?  _ YES  _ NO. 


It  ‘NO.*  list  discrepancies  observed  in  the  spaces  below 

- I 

. - . - . - . - . -  . . . . I 


4.  STOP  TIME: 
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USER  EVALUATION  QUESTIONNAIRE 


Parjicipani 


Introduction 

The  Portable  Maintenance  Aid  (PMA)  that  you  used  during  the  F/A<18  test  is  an  exampie 
of  how  maintenance  procedures  and  technical  data  may  be  deilvered  (or  next  generation  aircraft. 
Since  you  and  other  technicians  may  be  using  such  a  device  in  the  future,  your  feedback  on  the 
current  model  of  PMA  is  essential.  Accordingly,  this  questionnaire  was  designed  to  obtain  your 
opinions  about  the  design,  operation,  arid  usefulness  of  PMA. 

Specifically,  we  asK  you  to  evaluate  the  questionnaire  items  using  the  5-polnt  scale 
appearing  to  the  right  of  the  items.  Pate  each  Item  by  placing)  an  *X'  in  the  appropriate 
column.  We  encourage  you  to  respond  to  as  many  of  the  questionnaire  Items  as  possible  but 
recognize  that  there  may  be  some  items  you  cannot  evaluate  based  on  your  limited  experience 
with  PMA.  In  those  cases,  place  an  "X"  in  the  column  headed:  'Can't  Evaluate*. 

Section  1 :  Physical  Features  of  PMA 


Scale  Values 

Items 

Unsatis¬ 

factory 

Marginal 

Satisfactory 

Highly 

Satisfactory 

1 

Vi 

1 

S 

^  s 

LU 

1 .  Overall  weight  of  the  device. 

2.  Overall  size  (width  and  length)  of  the  device. 

3.  Overall  height  (thickness)  of  the  device. 

4.  Ease  of  positloninQ/repositionIng  PMA  at  the  worksite. 

5.  Ease  of  connecting  PMA  to  the  1553  bUs. 

7.  Location  of  keys. 

lEIIIIII^KSRlElIRSISIIIIIIIIilllllllllllllllllllllli^^ 
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Section  1:  Physical  Features  of  PMA  (cent.) 


Scale  Values 


9.  Response  time  after  key  press. 


10.  Appropriateness  of  function  keys. 


11.  Adequacy  of  screen  size  for  displaying  information. 


13.  Contrast  between  letters/graphics  and  background. 


14.  Amount  of  olare  on  display  screen, 


IS.  Key  pressure  resistence/sensitivlty, 


Section  2;  Software/Operational  Features  of  PMA 


Scale  Values 


16.  Spacing  of  Information  on  the  screen  (vs.  crowding). 


17.  Legibility  of  displayed  letters,  numbers,  and  words. 


IB.  Adequacy  of  organization/arrangement  of  information. 


19.  Adequacy  of  options  on  menus/fiinctlon  keys. 


20.  Adequacy  of  menu  organization. 


21.  Ease  of  using  men  us/I  unction  keys. 
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Section  2:  Sottware/Operation  Features  of  PMA  (cont.) 


Scale  Values 


22.  Ease  of  moving  cursor  with  arrow  keys. 


23.  Ease  cl  moving  cursor  by  pressing  number 
Keys. 


24.  Ease  of  moving  cursor  with  thumb  knob. 


25.  Ease  of  returning  to  appropriate  place  In  a  set  of 
procedures  after  branching  elsewhere  in  data  base. 


26.  Adequacy  of  Information  for  supporting  msdntonance 
tasks  (l.e.,  completeness,  accuracy,  relevance). 


27.  Appropriate  number  of  procedural  steps  per  screen. 


28.  Adequacy  of  PMA  tor  completing  supply 
requisitions,  VIDS/MAPs,  etc. 


1 1" 

Is  ^ 


Can'l 

Evahiafe 
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Section  2:  Software/Operatlonai  Featured  of  PMA  cont.) 


Scale  Values 

items 

Unsalis- 

lactoiy 

Marginal 

Satisfactory 

Highly 

Salisfaaory 

OulsiaiKJing 

Can't 

Evahiaie 

29.  Scrolling  function  availability. 

■ 

30.  Scroll  mode  (hard  Key). 

■ 

■ 

31 .  Scrolling  with  arrow  keys  and  SELECT  key. 

m 

32,  Availability  of  functions  on  soft  keys. 

■ 

33.  Menu  Item  names. 

■ 

m 

.H 

34.  Avallabllitvof  menu  functions. 

V 

35.  Cursor  visibility. 

35.  Legibility  of  graphics. 

B 

/ 

37.  Adequacy  of  detail  on  graphics. 

■ 

— 

38.  Ease  of  accessing  Locator  diagrams. 

■ 

Dl 

39.  Adeauacy  of  detail  orovlded  on  Locator  diaorams. 

v/' 

40.  Adequacy  of  wiring  diagrams. 
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Sections:  Comparative  Assessment 


This  section  of  the  questionnaire  deals  with  the  efficiency  and  effectiveness  of  PMA  compared  to  existing 
technical  manuals  lor  the  P/A*16  aircraft.  Note  that  the  words  listed  under  the  heading  scale  values 
have  changed.  Please  review  this  scale  carefully  before  rating  the  items  in  this  section.  To  avoid 
repetition  in  the  wording  of  the  items  contained  In  this  section,  begin  each  with  the  phrase: 


The  PMA  can  be  compared  to  P/A*1 8  technical  manual  Work  Packages  (WP)  In 
the  following  ways 


Scale  Values 

Items 

WP  SignificanUy 
Better 

WPSighiV 

Better 

No 

Difference 

PMASGghlly 

Better 

PMA  Significantly 
Better 

u 

LU 

41,  The  overall  time  and  effort  required  to  Obtain 
maintenance  Information. 

J 

42.  The  fatigue  you  experienced  when  using  It. 

/ 

43,  The  confusion  or  frustration  you  experienced  in 
obtaining  needed  technical  Information. 

y 

44,  The  overall  organization  and  arrangement  of 
technical  Information. 

■ 

1 

1 

u 

■ 

45  Obtaining  access  to  needed  technical  information. 

■ 

46.  The  method  of  presenting  technical  information. 

/ 

! 

47.  The  overall  oompletenesi,  accuracy,  and 
applicability  of  technical  information. 

48.  Supporting  maintenance  on  the  F/A-l 8  flight 
control  system. 

7 

Section  4:  General  Reactions  and  Comments 


The  spaces  below  are  provided  for  making  any  oomments,  complaints,  suggestions,  etc.  you  may  have 
regarding  the  current  model  of  the  PMA  device  (e.g.,  for  any  hern  rated  ‘unsatisfacto^*  or  'Marginal',  it 
would  be  helpful  to  know  why  it  received  that  rating).  Comments  may  be  continued  on  the  reverse  side  of 
this  page  it  necessary. 


I 


This  concludes  the  user  evaluation  questionnaire.  Your  assistance  in  providing  this  essential  inlotmation 
is  appreciated. 
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STRUCTURED  INTERVIEW  GUIDE 


Wh8t  we  will  be  talking  about  is  based  upon  the  degree  that  you  used  the 
following  PMA  features: 

•  Scrolling,  panning,  zooming 

•  Circuit  schematic  and  wiring  diagrams 

•  Switching  between  levels  of  detail 

•  On-line  HELP 

•  Cross-references  to  related  procedural  steps/technicat  information 

•  Electronic  form  filling  (e.g.,  for  supply  requistions,  VIDS/MAF,  etc.) 

I.  Specific  questions  for  PMA  users: 

1 .  Did  you  use  scrolling,  panning,  or  zooming?  If  YES,  then: 

a.  Were  these  features  useful? 

b.  On  a  scale  ol  1  to  5  with  1  being  very  easy,  and  5  being  very  ditticull,  how  easy  were  these  features  tc 
use? 

12  3  4  5 

!l  (]  []  []  (] 

Very  Easy  Very  DIflicuft 

c.  Can  you  add  to  tnat  or  give  an  example? 


2.  Did  you  use  circuit  schematic  or  wiring  diagrams?  If  YES,  then; 

a.  Was  enough  context  provided  on  the  diagrams  to  prevent  you  from  getting  'losf? 

b,  On  a  seals  ol  1  to  5  with  1  being  very  easy,  arvl  5  being  very  dilficult,  how  easy  were  schematc  and  wiring 


diagrams  to  use? 

1  2 

3 

4  5 

(1  11 

(1 

(1  1) 

Very  Easy 

Very  Difficult 

c.  Can  you  add  to  that  or  give  an  example? 


3.  Did  you  use  more  than  one  level  of  detail?  If  YES,  then: 


a.  Whch  level  ol  detail  was  most  uselul  lo  you?  Why? 

b.  On  a  scale  of  uo  5  with  t  being  very  useful,  and  5  being  not  uselul  ai  all  how  uselul  were  the  two  levels  o: 
detail? 

1  2  3  4  5 

|]  (I  []  (]  [] 

Very  Useful  Not  Useful  At  All 

c.  Are  two  levels  enough?  If  rtot,  how  many  levels  of  detail  should  be  provided? 


4 .  Did  you  use  on*llne  HELP?  If  YES,  then: 

a.  Did  on-line  HELP  provide  uselul  information? 

b^  On  a  scale  ol  1  lo  5  with  i  being  very  easy,  and  5  being  very  drtlicull,  how  easy  was  on-line  HELP  lo  use’ 

I  2  3  4  5 

II  11  11  II  0 

Very  Easy  Verj'  Difficult 

c.  Can  you  add  to  that  or  give  an  example? 


5.  Did  you  make  cross-references  to  other  procedural  sieps/technlcal  Information  In  the 
database?  If  YES,  then; 

a.  Was  the  cross-referenced  information  usetul'relevant  to  the  task  at  hand? 

b.  On  a  scale  of  1  lo  5  with  1  being  very  eat^y,  and  5  being  very  drllicjli,  how  easy  was  the  cross-reference 
feature  to  use? 

I  2  3  4  5 

II  (1  II  (1  11 

Very  Easy  Very  Difficult 

c.  Can  you  add  lo  that  or  give  an  example? 
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6,  Did  you  use  fault  Isolation  diagrams?  It  YES,  then: 


I 

I 


a.  Were  the  diagnostic  block  diagrams  organized  correctly? 

b.  On  a  scale  of  i  to  5  with  i  being  very  easy,  ar>d  5  beii^  very  difficult,  how  easy  were  the  diagnostic  block 
diagrams  to  use? 

1  2  3  4  5 

I)  II  II  11  (1 

Very  Easy  Very  Difficult 

c.  Did  the  shading  of  the  probable  faulty  conponents  rnake  sense? 

d .  Would  you  change  the  way  the  diagnostic  system  works?  How? 


7.  Did  you  use  the  Menu  System  and  Programmable  Function  Keys?  It  YES,  then: 

a.  What  data  access  method  did  you  use  more  frequently? 

b.  On  a  scale  of  1  to  5  with  1  being  very  easy,  and  5  being  very  diflicult,  how  easy  were  these  features  to 
use? 

1  2  3  4  5 

w 

Very  Easy  Very  Difficult 

c.  Which  method  did  you  like  better?  Why? 

cl.  Were  any  of  the  functions  difficult  to  understand?  Which  ones? 

e.  Would  you  rename  any  functions  to  make  them  easier  to  understand? 


8.  Did  you  use  the  number  system  and  cursor  for  data  selection?  If  YES,  then; 

a.  What  selection  method  did  you  use  more:  numbers  or  cursor? 

b.  On  a  scale  of  1  to  5  with  1  being  very  easy,  and  5  being  very  ditficult,  how  easy  were  these  features  to 
use? 

1  2  3  4  5 

(1  11  (1  (1  (1 

Very  Easy  Very  Difficult 

c.  Which  method  did  you  like  better?  Why? 

d .  What  method  did  you  prefer  for  cursor  movement;  ioysticK  or  arrow  keys?  Why? 


9.  For  th«  PMA  training  that  you  received: 


a.  Did  the  PMA  training  provide  enough  Inlormation  lor  you  to  work  etiedively  during  the  test?  Too 
muchToo  little? 

b.  Were  there  any  lunciions/leatures  that  were  not  trained  well  enough? 

c.  After  the  PMA  training,  would  you  feel  oomfortable  teaching  co-workers  how  to  use  the  PMA? 


10,  If  the  electronic  form  tilling  function  was  used: 

a.  Were  data  entries  that  were  made  'automatically*  by  PMA  complete  and  correct? 

b.  On  a  scale  of  t  to  5  with  t  being  very  easy,  ar>d  5  being  very  difticutt,  how  easy  was  the  electronic  form 
tilling  function  to  use? 

1  2  3  4  5 

I)  11  11  11  II 

Very  Easy  Very  Difticutt 

c.  Can  you  add  to  that? 


II.  Ask  the  following  "general"  questions  of  PMA  users: 

//Alt8rn9t§  MSking  qu»$tloni  til  and  ti2  until  you  hava  tha  thru  top  llkaa”  and  thru  worst 
"dlsllkaa”  llatsd.// 

1 1 .  What  did  you  IM  most  about  tha  PMA  ayttam? 

,1  SI : _ _ 


2nd; 


3rd:, 


1  2.  What  did  you  dislike  most  about  tha  PMA  tyatam? 
1st: _ 


2nd;. 


3rd:. 


13.  II  you  had  tha  choice,  would  you  prefer  to  use  an  automated  system  like  PMA  or  do 
you  prefer  to  use  conventional  (paper>based)  technical  manuals?  Why? 


14.  Are  there  any  other  maintenance  or  losistlcs  support  functions  that  you  think  an 
automated  system  like  PMA  eould/should  support?  If  so,  please  tell  me  what  these 
support  functions  might  Include. 
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15,  What  changes  would  you  recommend  to  Improve  the  automatic  PMA  system? 


16.  Can  you  foresee  any  problems  when  using  an  automated  system  Ilka  PMA  to  perform 
maintenance  duties  on  the  tllght'daek/tllght'llne?  If  so,  please  describe  to  me  what  these 
problems  would  be,  and  any  recommendations  you  may  have  tor  overcoming  them. 

-  I 

I 

- - - . 


Thank  you  very  much  for  your  partlclpatlcn.  You  have  provided  us  with  valuable 
performance  data  and  Information  on  the  usefulness  of  PMA  and  similar  systems. 
Please  wait  until  this  project  Is  finished  before  discussing  any  details  with  your 
friends.  They  may  be  taking  part  In  this  project  and  you  could  unduly  Influence  them. 

THANKS  AGAIN!! 
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APPENDIX  F 


SAMPLE  lETM/PMA  FRAMES 

VIDS/MAF  Showing  Discrepancy/Symptom 
Block  Dtagram  Screen 
Ranked  Actions  Screen 
Log  File  Screen 
Oosing  Actions  Screen 
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VIDS/MAF  Showing  Discrepancy/Symptom 


After  a  successful  sign  on,  a  representation  of  the  flavy  VlDS^MAF  form 
is  presented  (F»g.  7).  This  fonm  presents  Information  about  the  Job  that  the 
technician  r>eeds  to  ^  started.  The  technician  can  review  and  enter  Information 
required  for  the  form  as  r^ecessary.  The  man  number,  disaepancy,  bureau 
number,  date,  etc.,  are  Included  on  this  fonn.  At  the  end  of  the  rrvalrrtenarKe 
session,  information  wtH  be  automatically  filled  In  by  the  software.  This 

information  will  include  maintenan^' "action  taken,  performance  times  how 
malfurvhion  code,  etc. 
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Block  Diagram  Screen 


Jfi  or  WVLTIfWO**  DBrtJkT  OBOUr 


nCCOUUCKMO  ACnONi  tool 


(  Mill!**  ] 


iMtTmmaoM  o«BnjiY  otroop 


AMM 

u«  n* 

AX!  MMnr  1  UfMi  1  BB*I  1 

_ a. 

Figure  9.  Block  Diagram  Screen. 


The  first  troubleshooting  screen  presented  to  the  technician  will  be  an 
JMIS-DM  block  diagram  (Flo.  9).  This  erKoded  diagram  aids  the  technician 
by  conveying  the  toliowirg  rrformation:  the  ayslam  undergoing  troubleshooting; 
the  recommended  (or  selected)  maintenance  activity;  th»e  susf^ad  component 
or  eystem  (shaded  block);  the  coriponent  or  system  etfectw  by  the  activity 
(dark  border  around  block);  related  systems  (ellipse);  and  any  ootttponent  or 
system  which  have  been  removed  from  consideration  by  some  previous  action 
(diagonal  through  block).  Soft  keys  at  the  bottom  of  the  saeen  provide  access 
to  additional  information  the  technician  may  wish  to  see.  After  every  action 
(test  or  repair)  tfte  block  diagram  will  be  updated  to  reflect  the  current  status 
of  the  component  or  system. 
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Ranked  Actions  Screen 


After  reviewing  the  block  diMram,  the  technician  has  the  option  to  perform 
any  available  action,  not  just  rie  or^e  being  recommended  by  the  IMIS-DM 
Several  ranked  lists  are  provided  to  the  technician  to  aid  in  the  troubleshooting 
process.  One  such  list,  Ranked  Actions  (Fig.  10),  provides  an  interleaved  list 
of  additional  actions  the  technician  can  perform  other  than  the  recommended 
one.  Also  included  on  the  list  are  the  time  required  to  perform  the  action 
the  failure  probability  of  the  action,  and  the  availability  of  each  action.  The 
flexibility  built  Into  the  system  allows  the  user  to  take  full  advantage  of  their 
expen'enoe  and  knowledge  of  the  alrwaft  while  still  being  supported  by  the 


Figure  fO,  Ranked  Actions  Screen. 
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Log  FUe  Screen 


Page 


A  Log  File  is  maintained  during  the  session  (Fig.  14):  any  time  during  or 
after  the  session,  the  tedmidan  can  review  the  actions  they  performed  and 
their  status. 


1. 

2.  0 

3.  0 

4, 

3.  C 

ScM  Cool 

«or  1121  k 

■aor  mo  h 
Saplaoad  ( 
)UpUy»  Fu 

■  aquai  ta  Open 

1  equal  le  Open 
.DDI 

nellonal  Cheek  ta  equal  to  paae 

B 

OK 

IcanceU  r  w*'p  1 

OK 


CANCEL 


M*Ib  I 


Figure  14,  Log  Fie  Screen. 


138 


Page 


Closing  Actions  Screen 


Figure  IS.  Ciosing  Actions  Screen. 
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